Automated Host Collection Tool
Automated Host Collection Tool
Introduction
Prophet is an open-source automated collection and analysis tool developed by OneProCloud. It supports efficient data collection and detailed analysis for various environments, including physical servers, cloud hosts, and VMware. The tool is designed for cloud migration and disaster recovery planning. By automatically gathering key information from source hosts and comparing technical metrics, it ensures smooth migration or disaster recovery. Prophet has been successfully used in multiple real-world projects, proving its stable performance and reliable features.
More Details: View in Github
Submit New Issue: New Issue
Functions
Currently, Prophet consists of the following main features:
Network Host Scanning Scans the network for live hosts using the
nmapcommand, and attempts to analyze the basic information of the hosts through packet data.VMware Environment Information Collection Collects detailed host information through VMware API, including compute resources, storage, and network-related data, supporting host migration-related analysis.
Linux Host Information Collection Uses Ansible to gather detailed information from Linux hosts, including compute resources, storage, and network-related data.
Windows Host Information Collection Collects detailed information from Windows hosts using the Windows WMI interface, including compute resources, storage, and network-related data.
Data Packaging and Anonymization Packages and compresses the collected data in YAML format, and performs data anonymization by removing sensitive user information.
Data Analysis and Technical Conclusions Analyzes the collected data to generate a comprehensive technical research report, providing a reliable foundation for migration and disaster recovery planning.
Installation Guide
Recommended Environment:
To ensure the best performance of the Prophet tool, use the following environment:
- Operating System: Ubuntu 20.04 and above
- Container Support: Docker (Ubuntu's built-in version)
Installation Process:
- Install Docker: On Ubuntu, directly install Docker using the system's package manager:
sudo apt update
sudo apt install -y docker.io- Start and Verify Docker Service: After installation, start the Docker service, enable it to start on boot, and check its status:
sudo systemctl start docker
sudo systemctl enable docker
sudo systemctl status dockerVerify Docker installation:
sudo docker --versionAfter installation, you can run the Prophet tool in a containerized environment.
Get Prophet Container
Download Prophet Container Image
Use the following command to download the latest version of Prophet from the container image repository:
docker pull registry.cn-beijing.aliyuncs.com/oneprocloud-opensource/cloud-discovery-prophet:latestRun Prophet Container Service
Use the following command to start the Prophet container service:
docker run -d -ti --name prophet registry.cn-beijing.aliyuncs.com/oneprocloud-opensource/cloud-discovery-prophet:latestAccess Prophet Container
After the container starts, you can enter the container using the following command for further operations:
docker exec -ti prophet bashUse Prophet CLI
After entering the container, you can use the Prophet CLI tool with commands supporting
scan,collect, andreportoperations:prophet-cli {scan,collect,report}
Follow these steps to complete the deployment and use of the Prophet container service.
User Guide
Basic Usage Process
- Scan the specified IP address range.
- In the CSV file of the scan results, provide authentication information for the hosts you want to retrieve details for.
- Perform batch collection.
- Analyze and obtain results.
Function 1: Scan All Running Instances on the Network
Function Description
Discover live hosts within a specific network segment through network scanning and record the findings. This information can be used as input for more detailed information collection in subsequent steps. After the scan is complete, a scan_results.csv file will be automatically generated at the specified path to store the information.
Note: To prevent significant pressure on the production environment, the scan is performed in a single-process mode, resulting in a slower scanning progress. It has been estimated that scanning a subnet with a mask of 24 requires approximately 30 minutes.
usage: prophet-cli scan [-h] --host HOST [--arg ARG] --output-path OUTPUT_PATH
[--report-name REPORT_NAME]
optional arguments:
-h, --help show this help message and exit
--host HOST Input host, example: 192.168.10.0/24, 192.168.10.1-2
--arg ARG Arguments for nmap, for more detailed, please check
nmap document
--output-path OUTPUT_PATH
Generate initial host report path
--report-name REPORT_NAME
Scan report csv nameExample 1: Obtain Subnet Hosts
Scan all live hosts in the 192.168.10.0/24 subnet and generate a CSV file in the /tmp directory.
prophet-cli scan --host 192.168.10.0/24 --output-path /tmp/Example 2: Obtain Hosts in a Specified IP Range
Scan all live hosts in the range 192.168.10.2 to 192.168.10.50 and generate a CSV file in the /tmp directory.
prophet-cli scan --host 192.168.10.2-50 --output-path /tmp/CSV Structure Explanation
| Field Name | Field Description |
|---|---|
| hostname | Hostname, can be empty |
| ip | User IP address, mandatory |
| username | Username; for VMware, it is the username for ESXi or vCenter |
| password | Password; for VMware, it is the password for ESXi or vCenter |
| ssh_port | For Linux, this field represents the SSH port; for VMware ESXi or vCenter, it is the connection port, defaulting to 443; for Windows, it is empty by default |
| key_path | If using key-based login, specify the absolute path to the key; otherwise, it is empty |
| mac | Host MAC address, can be empty |
| vendor | Vendor, can be empty; if it is a virtual machine running on VMware, it will be VMware |
| check_status | Whether detailed information collection is needed; set to "check" if required, otherwise, the tool will automatically skip |
| os | Operating system type; currently supported types are: Linux/Windows/VMware; case-sensitive |
| version | Operating system version, can be empty |
| tcp_ports | Open external ports, can be empty |
| do_status | Detailed information collection status, indicating whether collection is complete or failed; default is empty |
Function 2: Detailed Information Collection
Function Description
After users input authentication information into the template, further detailed scanning is performed.
Note:
- For hosts to be scanned, set the
check_statusto "check"; otherwise, no check will be performed. - For VMware virtual machines, scanning will only be done through the associated ESXi host.
- For Windows hosts, scanning requires the Administrator user.
- Successfully collected hosts will not undergo collection again upon subsequent script runs unless the user specifies the
force-checkparameter. - Hosts that fail in collection will be reattempted in the next collection.
- In the final generated compressed package, all sensitive information related to user authentication has been removed.
- (Stable) The VMware collection part is currently stable.
- (Testing) The Linux and Windows collection parts are still in testing.
usage: prophet-cli collect [-h] --host-file HOST_FILE --output-path
OUTPUT_PATH [-f] [--package-name PACKAGE_NAME]
optional arguments:
-h, --help show this help message and exit
--host-file HOST_FILE
Host file which generated by network scan
--output-path OUTPUT_PATH
Output path for batch collection
-f, --force-check Force check all hosts
--package-name PACKAGE_NAME
Prefix name for host collection packageExample: Execute Collection
First, update the authentication information for the hosts to be collected in the generated scan_results.csv. The project's examples directory includes a sample scan_results.csv.
| hostname | ip | username | password | ssh_port | key_path | mac | vendor | check_status | os | version | tcp_ports | do_status |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 192.168.10.2 | administrator@vsphere.local | your_vcenter_password | 00:50:56:9f:03:f0 | check | Windows | Microsoft Windows 7 or Windows Server 2012 | 80,88,135,139,389,443,445,514,636,2020,3389,8088,9009,9090 | |||||
| 192.168.10.6 | root | your_esxi_password | 443 | 0c:c4:7a:e5:5d:20 | check | VMware | VMware ESXi Server 4.1 | 22,80,427,443,902,5988,5989,8000,8080,8100,8300 | ||||
| 192.168.10.185 | Administrator | your_windows_password | 00:50:56:9a:6d:2e | check | Windows | Microsoft Windows Server 2003 SP1 or SP2 | 135,139,445,1028,3389 | |||||
| 192.168.10.201 | root | your_linux_password | 22 | ac:1f:6b:27:7f:e4 | check | Linux | Linux 2.6.32 - 3.9 | 22,80,3306,4567,5000,5900,5901,5902,5903,5904,5906,5907,5910,5911,5915,8022,8080,9100 |
prophet-cli collect --host-file /tmp/scan_hosts.csv --output-path /tmpCollection Result Explanation
Collection Directory Structure
hosts_collection
|-- LINUX -> Linux Host Collection Information
|-- VMWARE -> VMWare Host Collection Information
|-- WINDOWS -> Windows Host Collection Information
|-- prophet.log -> Log during the collection process, useful for analyzing unknown scenarios
|-- scan_hosts.csv -> File containing information on collected hosts, including open port detailsAdditionally, a hosts_collection_xxxxxxx.zip file will be generated in the output directory. This file is the final compressed file used for analysis.
Function 3: Analyze and Generate Reports
Function Description
Analyze the collected results and generate the final migration report. This section can be expanded based on specific requirements.
usage: prophet-cli report [-h] --package-file PACKAGE_FILE --output-path
OUTPUT_PATH [--report-name REPORT_NAME] [--clean]
optional arguments:
-h, --help show this help message and exit
--package-file PACKAGE_FILE
Investigate package file which is genreated by
prophet-collect
--output-path OUTPUT_PATH
Generate report path
--report-name REPORT_NAME
Generate report name
--clean Clean temp work dir or not, by default is notExample: Analyze and Generate Report
prophet-cli -d -v report --package-file /tmp/hosts_collection_20211215202459.zip --output-path /tmpExample: Analyze Report
| Platform Type | Hostname | VMware Hostname | IP | Mac | Operating System Type | Operating System Version | Operating System Bit | Operating System Kernel Version | Boot Method | CPU | CPU Cores | Memory | Total Memory (GB) | Free Memory (GB) | Disk Count | Total Disk Capacity (GB) | Disk Information | Partition Information | Network Card Count | Network Card Information | Virtualization Type | Virtualization Version | ESXi Server |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Physical | compute201 | 192.168.10.201 | ac:1f:6b:27:7f:e4 | CentOS | 7.5.1804 | x86_64 | 3.10.0-862.el7.x86_64 | bios | Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz | 40 | 62.65 | 5.35 | 7 | 5467004.78 | sda|56266.78|ATA|SuperMicro SSD sdb|213212.97|ATA|INTEL SSDSC2BB24 sdc|1039505.00|SEAGATE|ST1200MM0007 sdd|1039505.00|SEAGATE|ST1200MM0007 sde|1039505.00|SEAGATE|ST1200MM0007 sdf|1039505.00|TOSHIBA|AL15SEB120NY sdg|1039505.00|TOSHIBA|AL15SEB120NY | /dev/sda2|62245572608|58963365888|0.95|xfs /dev/sda1|1063256064|922746880|0.87|xfs /dev/sdc1|1199655976960|1141407465472|0.95|xfs | 4 | eno1|0c:c4:7a:a5:29:3a|True|1500|10000|10.0.100.201|255.255.255.0|10.0.100.0|10.0.100.255|fe80::ec4:7aff:fea5:293a ens2f3|ac:1f:6b:27:7f:e7|True|1500|1000|None|None|None|None|fe80::ae1f:6bff:fe27:7fe7 ens2f0|ac:1f:6b:27:7f:e4|True|1500|1000|192.168.10.201|255.255.255.0|192.168.10.0|192.168.10.255|192.168.10.1|fe80::ae1f:6bff:fe27:7fe4 ens2f1|ac:1f:6b:27:7f:e5|True|1500|1000|172.16.100.201|255.255.255.0|172.16.100.0|172.16.100.255|fe80::ae1f:6bff:fe27:7fe5 | |||||
| Physical | compute202 | 192.168.10.202 | 0c:c4:7a:e5:5c:f8 | CentOS | 7.5.1804 | x86_64 | 3.10.0-862.el7.x86_64 | bios | Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz | 40 | 62.65 | 13.27 | 7 | 5467004.78 | sda|56266.78|ATA|SuperMicro SSD sdb|213212.97|ATA|INTEL SSDSC2BB24 sdc|1039505.00|SEAGATE|ST1200MM0007 sdd|1039505.00|SEAGATE|ST1200MM0007 sde|1039505.00|SEAGATE|ST1200MM0007 sdf|1039505.00|TOSHIBA|AL15SEB120NY sdg|1039505.00|TOSHIBA|AL15SEB120NY | /dev/sda2|62245572608|52926242816|0.85|xfs /dev/sda1|1063256064|922746880|0.87|xfs | 4 | eno1|0c:c4:7a:a5:2e:d0|True|1500|10000|10.0.100.202|255.255.255.0|10.0.100.0|10.0.100.255|fe80::ec4:7aff:fea5:2ed0 ens2f3|0c:c4:7a:e5:5c:fb|True|1500|1000|None|None|None|None|fe80::ec4:7aff:fee5:5cfb ens2f0|0c:c4:7a:e5:5c:f8|True|1500|1000|192.168.10.202|255.255.255.0|192.168.10.0|192.168.10.255|192.168.10.1|fe80::ec4:7aff:fee5:5cf8 ens2f1|0c:c4:7a:e5:5c:f9|True|1500|1000|172.16.100.202|255.255.255.0|172.16.100.0|172.16.100.255|fe80::ec4:7aff:fee5:5cf9 | |||||
| VMware | node01 | 00:50:56:9a:49:b7 | CentOS 4/5/6/7 | CentOS 4/5/6/7 | 64-bit | efi | Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz | 4 | 8192.00 | 1 | 51200.00 | [10.3-4T-5] centos7.4_node_139/centos7.4_node_139.vmdk|51200.00 | 1 | VM Network|00:50:56:9a:49:b7 | VMware ESX Server | VMware ESXi 6.0.0 build-2715440 | 192.168.10.3 | ||||||
| VMware | master03 | 00:50:56:9a:63:a0 | CentOS 4/5/6/7 | CentOS 4/5/6/7 | 64-bit | efi | Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz | 4 | 4096.00 | 1 | 51200.00 | [10.3-4T-5] centos7.2_132/centos7.2_132.vmdk|51200.00 | 1 | VM Network|00:50:56:9a:63:a0 | VMware ESX Server | VMware ESXi 6.0.0 build-2715440 | 192.168.10.3 | ||||||
| VMware | master02 | 00:50:56:9a:06:ad | CentOS 4/5/6/7 | CentOS 4/5/6/7 | 64-bit | efi | Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz | 4 | 4096.00 | 1 | 51200.00 | [10.3-4T-5] centos7.3_131/centos7.3_131.vmdk|51200.00 | 1 | VM Network|00:50:56:9a:06:ad | VMware ESX Server | VMware ESXi 6.0.0 build-2715440 | 192.168.10.3 | ||||||
| Physical | COMPUTER-PC | 192.168.10.62 | 00:0c:29:9a:59:73 | Microsoft Windows 7 旗舰版 | Microsoft Windows 7 旗舰版 | 64-bit | 6.1.7600 | bios | Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz | 4 | Physical Memory | 8191.55 | 4.83 | 2 | 255996.72 | 0|51199.34|VMware Virtual disk SCSI Disk Device|VMware Virtual disk SCSI Disk Device 1|204797.37|VMware Virtual disk SCSI Disk Device|VMware Virtual disk SCSI Disk Device | C:|53580132352|4455071744|0.08|NTFS D:|109887614976|109790175232|1.0|NTFS E:|104857595904|104760311808|1.0|NTFS | 1 | [00000007] Intel(R) PRO/1000 MT Network Connection|00:0c:29:9a:59:73|192.168.10.1|192.168.10.62|255.255.255.0 |