XMap initial release

XMap is a fast network scanner designed for performing Internet-wide IPv6 & IPv4 network research scanning.

.clang-format

@@ -0,0 +1,22 @@
+BasedOnStyle: LLVM
+PointerAlignment: Right
+IndentWidth: 4
+MaxEmptyLinesToKeep: 1
+ObjCSpaceAfterProperty: true
+ObjCBlockIndentWidth: 4
+AllowShortFunctionsOnASingleLine: true
+AllowShortIfStatementsOnASingleLine: true
+AlignTrailingComments: true
+SpacesInSquareBrackets: false
+SpacesInParentheses : false
+AlignConsecutiveDeclarations: true
+AlignConsecutiveAssignments: true
+SpaceBeforeAssignmentOperators: true
+SpacesInContainerLiterals: true
+IndentWrappedFunctionNames: true
+KeepEmptyLinesAtTheStartOfBlocks: true
+BreakConstructorInitializersBeforeComma: true
+AllowAllParametersOfDeclarationOnNextLine: true
+SpaceAfterCStyleCast: true
+TabWidth: 4
+UseTab: Never

.gitignore

@@ -0,0 +1,23 @@
+
+.idea
+.gitattributes
+.DS_Store
+cmake-build-debug
+cmake_install.cmake
+CMakeCache.txt
+CMakeFiles
+CPack*
+Makefile
+!examples/forge-socket/Makefile
+lib/libxmaplib.a
+src/lexer.c
+src/lexer.h
+src/parser.c
+src/parser.h
+src/xmap
+src/xopt.c
+src/xopt.ggo
+src/xopt.h
+install_manifest.txt
+Testing
+*.cbp

.travis.yml

@@ -0,0 +1,42 @@
+language: c
+os: linux
+dist: xenial
+addons:
+  apt:
+    packages:
+      - build-essential
+      - cmake
+      - libgmp3-dev
+      - gengetopt
+      - libpcap-dev
+      - flex
+      - byacc
+      - libjson-c-dev
+      - pkg-config
+      - libunistring-dev
+      - libsasl2-dev
+      - python-dev
+      - python-pip
+      - libhiredis-dev
+before_install:
+  - pip install --user sh
+  - export PATH=/opt/bin:$PATH
+before_script:
+  - wget -O - https://apt.llvm.org/llvm-snapshot.gpg.key|sudo apt-key add -
+  - echo "deb http://apt.llvm.org/trusty/ llvm-toolchain-trusty-6.0 main" | sudo tee -a /etc/apt/sources.list
+  - sudo add-apt-repository -y ppa:ubuntu-toolchain-r/test
+  - sudo apt-get update -qq
+  - sudo apt-get install libstdc++-4.9-dev libllvm6.0 clang-format-6.0
+script:
+  - mkdir __build
+  - cd __build
+  - cmake -DENABLE_DEVELOPMENT=ON -DWITH_JSON=ON -DWITH_REDIS=ON $TRAVIS_BUILD_DIR
+  - make
+  - cd $TRAVIS_BUILD_DIR
+  - python ./scripts/check_manfile.py
+compiler: clang
+notifications:
+  email:
+    - [email protected]
+  slack:
+    secure: WaFUBQz19oVEZL5lu3+GCU9q/e45POGYgH27jlbrBOtIUebIey219oSDC2G+5sG1L547nVlkrVa95L7C6sT5RP3D6aT8gWI0aDubaGjp4Wc1crau0LepxfRwpLOG/cYhQsaG31R/c5a3JIvgasV9P70bPyY9fhKbVxD7LbkWF7v7GB3jaJMjBZHHqWCXuzBoqowkqZARI13AAD7wbJJpkw8C/Tkw0xx1BvQsH+5DpPgd/0UfHBMotT4SMP2oHWUAjwL1KaxUxn/a4IpYTxMp4vHrTmtj7GxsQGPf2QQ80f2s5fCzd5veyqZfo3sMRbu/LgomwHTPsIhK8l3t4EXwSuxxmwQFqrEl+wQ0qM764cgKmIoqoeHMPt9RBQl9x7j8wPw46n5S/bNpTqAS2w3cuqpkkVqdD4wEST9rg4p4ERnJgvLvmdxAHFzl72qVwXwb+KhTEzdDgS/1OjdBGJDjLe6tBFdvCrNd6LQHx3NfDCxbQG3FyALIGMDsxDP3SLS0M8zkkCkEeShIrGDlshQQc8dL3YjTY08euTauaC6Esm6fZy4dWRM9EuRwYXFwwpyXTs/HWkIm+DUEk1IU2/m4YLb5IXnTQ+wjAPB2+oZqIZPDK6izo6tE3tCWWZuCr67Z8BQsM3oIAY98LDy3Nu34jJOuOjq0fwEoSEEgNoeB9zc=

10gigE.md

@@ -0,0 +1,59 @@
+10GigE (Zippier) XMap
+===========
+
+It is possible to build XMap to run at 95% of 10 GigE linespeed, sending over 14
+million packets per second. This requires a compatible Intel 10 Gbps Ethernet
+NIC and Linux.
+
+### Prerequisites
+
+0. A working XMap development environment (see [INSTALL.md](https://github.com/idealeer/xmap/blob/master/README.md))
+   1. A [PF_RING ZC](http://www.ntop.org/products/pf_ring/pf_ring-zc-zero-copy/) license from ntop.
+2. PF_RING ZC headers and kernel module
+3. A 10 Gbps NIC with compatible "PF_RING-aware" drivers
+4. A Linux (not BSD or Mac) installation
+4. For best results, a computer with at least 8 *physical* cores on the same NUMA node.
+6. libnuma (`sudo apt-get install libnuma-dev`)
+
+### PF_RING Installation
+
+Coming soon.
+
+### Building
+
+Most build errors are due to incorrectly building or installing PF_RING. Make
+sure you have build the drivers, the kernel module, and the userland library, as
+well as install the headers and kernel module to the correct locations.
+
+The PF_RING `make install` command might not copy `pfring_zc.h` to
+`/usr/include`, in which case manually install the file and set permissions
+correctly.
+
+To build navigate to the root of the repository and run:
+
+```
+$ cmake -DWITH_PFRING=ON -DENABLE_DEVELOPMENT=OFF .
+$ make
+```
+
+### Running
+
+You'll have to carefully select the number of threads to use, as well as specify
+as zero-copy interface, e.g. `zc:eth1`. Use the `--cores` option to pick which
+cores to pin to. Make sure to pin to different physical cores, and note that
+some machines interleave physical and "virtual" cores.
+```
+$ sudo ./src/xmap -p 80 -i zc:eth7 -o output.csv -T 5
+```
+
+### Considerations
+
+**DO NOT TAKE THIS LIGHTLY!**
+
+Running XMap at 10Gbps hits every IPv4 prefix /16 on the Internet over 200 times a second.
+Even if you have a large source IP range to scan from, it's very obvious that
+you're scanning. As always, follow scanning best practices, honor blocklist
+requests, and signal benign/research intent via domain names and websites on
+your scan IPs.
+
+Remember, you're sending a lot of traffic.

AUTHORS

@@ -0,0 +1,10 @@
+Zakir Durumeric <[email protected]>
+David Adrian <[email protected]>
+Eric Wustrow <[email protected]>
+J. Alex Halderman <[email protected]>
+Paul Pearce <[email protected]>
+Ariana Mirian <[email protected]>
+HD Moore <[email protected]>
+Oliver Gasser <[email protected]>
+Quirin Scheitle <[email protected]>
+Xiang Li <[email protected]>

CHANGELOG.md

@@ -0,0 +1,2 @@
+# 1.0.0 2021-4-2
+* Initial public release.

CMakeLists.txt

@@ -0,0 +1,175 @@
+cmake_minimum_required(VERSION 2.8.12 FATAL_ERROR)
+project(XMAP C)
+set(XMAP_VERSION DEVELOPMENT)    # Change DEVELOPMENT to version number for release
+
+option(WITH_JSON "Build with support for JSON" ON)
+option(WITH_REDIS "Build with support for Redis DB" OFF)
+option(ENABLE_DEVELOPMENT "Enable development specific compiler and linker flags" OFF)
+option(RESPECT_INSTALL_PREFIX_CONFIG "Respect CMAKE_INSTALL_PREFIX for /etc" OFF)
+option(WITH_WERROR "Build with -Werror" OFF)
+option(WITH_PFRING "Build with PF_RING ZC for send (10 GigE)" OFF)
+option(ENABLE_LOG_TRACE "Enable log trace messages" OFF)
+option(FORCE_CONF_INSTALL "Overwrites existing configuration files at install" OFF)
+
+if ("${CMAKE_CXX_COMPILER_ID}" STREQUAL "Clang")
+    set(USING_CLANG "YES")
+else ()
+    set(USING_GCC "YES")
+endif ()
+
+if ("${CMAKE_SYSTEM_NAME}" MATCHES "FreeBSD" OR "${CMAKE_SYSTEM_NAME}" MATCHES "NetBSD" OR "${CMAKE_SYSTEM_NAME}" MATCHES "DragonFly")
+    set(BSD "YES")
+endif ()
+
+if ("${CMAKE_SYSTEM_NAME}" MATCHES "NetBSD")
+    set(NetBSD "YES")
+endif ()
+
+# Hardening and warnings for building with gcc
+# Maybe add -Wstrict-prototypes -Wmissing-prototypes -Wmissing-declarations
+set(GCCWARNINGS
+        "-Wall -Wformat=0 -Wno-format-nonliteral"
+        "-pedantic -fno-strict-aliasing"
+        "-Wextra"
+        "-Wfloat-equal -Wundef -Wwrite-strings -Wredundant-decls"
+        "-Wnested-externs -Wbad-function-cast -Winit-self"
+        "-Wmissing-noreturn"
+        "-Wstack-protector"
+        )
+
+# Fix line breaks
+string(REPLACE ";" " " GCCWARNINGS "${GCCWARNINGS}")
+
+if (WITH_WERROR)
+    set(GCCWARNINGS "${GCCWARNINGS} -Werror")
+endif ()
+
+if (ENABLE_DEVELOPMENT)
+    set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -ggdb")
+    set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -g")
+else ()
+    # Hardening and optimizations for building with gcc
+    set(GCCHARDENING "-U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-all -fwrapv -fPIC --param ssp-buffer-size=1")
+    if (NOT APPLE AND NOT BSD)
+        set(LDHARDENING "-z relro -z now")
+    else ()
+        set(LDHARDENING "")
+    endif ()
+
+    set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} ${GCCHARDENING} -O2")
+    set(CMAKE_EXE_LINKER_FLAGS "${LDHARDENING} ${CMAKE_EXE_LINKER_FLAGS}")
+endif ()
+
+set(CMAKE_C_FLAGS "${GCCWARNINGS} ${CMAKE_C_FLAGS}")
+
+if (ENABLE_LOG_TRACE)
+    add_definitions("-DDEBUG")
+endif ()
+
+if (WITH_JSON)
+    include(FindPkgConfig)
+    pkg_check_modules(JSON json-c)
+    if (JSON_FOUND)
+        include_directories(${JSON_INCLUDE_DIRS})
+    else ()
+        message(FATAL_ERROR "Did not find libjson")
+    endif ()
+    add_definitions("-DJSON")
+    string(REPLACE ";" " " JSON_CFLAGS "${JSON_CFLAGS}")
+    set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} ${JSON_CFLAGS}")
+endif ()
+
+if (WITH_REDIS)
+    set(REDIS_LIBS hiredis)
+    add_definitions("-DREDIS")
+endif ()
+
+if (WITH_PFRING)
+    add_definitions("-DPFRING")
+    set(PFRING_LIBRARIES pfring rt numa)
+endif ()
+
+# Standard FLAGS
+set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -std=gnu99")
+if (NOT APPLE)
+    set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -pthread")
+endif ()
+
+# Set up OS-specific include directories
+if (APPLE)
+    if (EXISTS /opt/local/include)
+        include_directories(/opt/local/include)
+    endif ()
+    if (EXISTS /opt/local/lib)
+        link_directories(/opt/local/lib)
+    endif ()
+    if (EXISTS /usr/local/include)
+        include_directories(/usr/local/include)
+    endif ()
+    if (EXISTS /usr/local/lib)
+        link_directories(/usr/local/lib)
+    endif ()
+endif ()
+
+if (BSD)
+    include_directories(/usr/local/include)
+    link_directories(/usr/local/lib)
+endif ()
+
+if (NetBSD)
+    include_directories(/usr/pkg/include)
+    link_directories(/usr/pkg/lib)
+endif ()
+
+add_subdirectory("lib")
+add_subdirectory("src")
+
+# Install conf files
+if (RESPECT_INSTALL_PREFIX_CONFIG)
+    set(CONFIG_DESTINATION "etc/xmap")
+else ()
+    set(CONFIG_DESTINATION "/etc/xmap")
+endif ()
+
+FILE(GLOB CONF_FILES "${PROJECT_SOURCE_DIR}/conf/*")
+
+message(STATUS "Default XMap configuration file location is /etc/xmap")
+foreach (EACH_CONF ${CONF_FILES})
+    get_filename_component(CONF_BASENAME ${EACH_CONF} NAME)
+    message(STATUS "Checking if ${CONF_BASENAME} exists there...")
+    if (NOT EXISTS "/etc/xmap/${CONF_BASENAME}")
+        install(FILES ${EACH_CONF} DESTINATION ${CONFIG_DESTINATION})
+    elseif (FORCE_CONF_INSTALL)
+        message(WARNING "FORCE_CONF_INSTALL will overwrite any exsiting configuration files")
+        install(FILES ${EACH_CONF} DESTINATION ${CONFIG_DESTINATION})
+    else ()
+        message(WARNING "Existing configuration file detected at /etc/xmap/${CONF_BASENAME}, ${CONF_BASENAME} from sources will NOT be installed. Please check and install manually!")
+    endif ()
+endforeach ()
+
+# Uninstall
+add_custom_target(uninstall
+        ${CMAKE_COMMAND} -P "${CMAKE_SOURCE_DIR}/scripts/uninstall.cmake"
+        )
+
+# Allow Debian Packaging
+include(InstallRequiredSystemLibraries)
+
+set(CPACK_SET_DESTDIR "on")
+set(CPACK_PACKAGING_INSTALL_PREFIX "/tmp")
+set(CPACK_GENERATOR "DEB")
+
+set(${VERSION} CPACK_DEBIAN_PACKAGE_VERSION)
+set(CPACK_DEBIAN_PACKAGE_PRIORITY "optional")
+set(CPACK_DEBIAN_PACKAGE_SECTION "network")
+set(CPACK_DEBIAN_ARCHITECTURE ${CMAKE_SYSTEM_PROCESSOR})
+set(CPACK_DEBIAN_PACKAGE_DEPENDS "libc6 (>= 2.1.3), libgmp10, libpcap0.8, libjson-c-dev")
+
+set(CPACK_PACKAGE_DESCRIPTION "Internet-wide IPv6 & IPv4 Network Scanner")
+set(CPACK_PACKAGE_DESCRIPTION_SUMMARY "XMap is an open-source network scanner that enables researchers to perform Internet-wide IPv6 & IPv4 network research scanning efficiently. XMap is reimplemented and improved thoroughly from ZMap and is fully compatible with ZMap, armed with the " 5 minutes" probing speed and novel IPv6 scanning techniques. XMap can be used to conduct large-scale IPv6 network scanning for device discovery and security assessments and help us gain more insights into the IPv6 networks.")
+set(CPACK_PACKAGE_CONTACT "Xiang Li <[email protected]>")
+set(CPACK_PACKAGE_FILE_NAME "${CMAKE_PROJECT_NAME}_${VERSION}_${CPACK_DEBIAN_ARCHITECTURE}")
+
+set(CPACK_COMPONENTS_ALL Libraries ApplicationData)
+
+include(CPack)

CONTRIBUTING.md

@@ -0,0 +1,35 @@
+Contributing to XMap
+====================
+
+XMap accepts contributions in the form of issues and pull requests. In either
+case, before posting please [search](https://github.com/idealeer/xmap/issues) to see
+if your change or bug report has been addressed previously.
+
+[INSTALL](INSTALL.md#building-from-source) provides guidance on building XMap from source.
+
+Developing
+----------
+
+- XMap code follows the [Linux kernel style guide][kernelguide]. We mantain [a
+  configuration file](/.clang-format) for `clang-format` that applies this
+  style. You can use the [format.sh](/format.sh) script to apply this style.
+
+- Before submitting a PR, please rebase/squash your commits down to a single
+  commit. Follow these [commit message guidelines][guidelines], especially with
+  regard to formatting.
+
+Reviewing
+---------
+
+- All commits must be reviewed in the form of a pull request by a XMap
+  maintainer. This usually means @idealeer.
+
+- All pull-requests should be squash-merged into master.
+
+- When squash-merging, put the PR number in the commit title. GitHub does this
+  automatically in the web interface.  Condense the commit messages down to a
+  single message; often this can just be the commit message from the first
+  commit in a PR. Follow the commit formatting guidelines [here][guidelines].
+
+[kernelguide]: https://www.kernel.org/doc/Documentation/process/coding-style.rst
+[guidelines]: https://github.com/torvalds/subsurface-for-dirk/blob/master/README#L92