Data collection
with Cortex
Efficiently collect and distribute vast amounts of (data) files from a wide variety of protocols at revolutionary speeds. Start dispatching data with our high-performance managed data distribution tool.
Cortex Data Dispatcher
The Cortex stack is a set of components for high-performance managed data collection and distribution.
The components in the Cortex stack are designed with properties to allow efficient management. Components and protocols can be combined to allow for a wide range of distribution scenarios. It consists of an advanced dispatch module and an (S)FTP scanner. The combined Cortex modules allow your system to push and pull data (pub/sub, also known as publishing and subscribing). The complete stack makes it the perfect data collection system for all your data collection needs.
High-Performance
Cortex is written in the high-performance systems programming language Rust, which means it is blazingly fast and memory-efficient. By making full use of parallelism on modern CPUs, the Cortex Stack is almost always bound by the performance of the network interfaces and not by the system.
Cortex integrates easily within your existing architecture and supports practically any source format. Some examples are;
- Cloud storage providers (Google Cloud, OneDrive, AWS S3, Dropbox, and many more)
- CORBA
- (s)FTP, HTTP(S)
- rsync
Cortex features
Configuration per data stream
The data retention and data distribution are configurable per data stream. It is also possible to set up both scheduled and continuous data retrieval and publication.
Metrics for networks
Cortex supports publishing metrics to tools like Prometheus out-of-the-box and includes default dashboards for monitoring and alerting purposes.
Operational Logging
Detailed logging is supported for all data streams, so anomalies can be investigated with ease. Logs can be shipped to standard logging facilities such as Syslog. journald and other log servers.
Audit Logging Support
Audit logging is enabled by default for all the incoming and outgoing data. This ensures the data and the access to it remains traceable.
Data Quality
Data Quality is guaranteed by making use of hashes. This allows for the de-duplication of data, ensuring the data is never altered or lost. Even at later stages e.g. the subscriber side of the data, this hash can be used to verify integrity.
SUPPORTED DATA DISPATCHER PROTOCOLS AND PROVIDERS
Cortex can distribute all data sources. We support a wide range of protocols for collecting and distributing data that can be freely combined;
- Pull collecting: HTTP, HTTPS, FTP, SFTP, SMB, NFS
- Push collecting: FTP, SFTP, SMB, NFS
- Pull distributing: HTTP, HTTPS, FTP, SFTP, SMB, NFS
- Push distributing: FTP, SFTP, SMB, NFS
No matter the vast amounts of data that need distributing, Cortex has already proven itself to dispatch vast amounts of data over multiple cross-domain networks with a variety of configurations per data stream.
| Some supported File Format examples |
| .xml |
| .csv |
| .xlsx |
| .xls |
| .dat |
| .txt |
| .bin |
| .png |
| .jpg |
| .mp4 |
Not convinced yet?
Send us sample data to process and be surprised!