Our Central Government client are recruiting for a specialist with specific and significant experience in Apache Kafka.
Our clients Kafka system (consisting of 46 x Apache Kafka hosts) and its subcomponents (MirrorMaker, Zookeeper, Broker, Replication, Clusters) was never fully documented by its implementers. The implementers are no longer with the organisation. This system is pivotal to an efficient and stable log ingestion platform for the SIEM and the SOC operation. The system is also not monitored from a functional capacity i.e. it's messaging processing, lag, replication, throughput etc.
There is a critical need for this system to have a comprehensive health check from End-to-End, by experts in this field. If we conduct the proposed Health check and get a comprehensive report and recommendations from this consultation, we can look to implement these, and introduce a "Control Centre" function. This will then give RSO the confidence that it has a well baselined system, and it is in a "fit state" and then implement a central Monitoring, Management and configuration element.
The investment and time in Elastic, without giving due diligence and attention to the core elements of the Data Log Shipping pipeline.
Further to the Health & System check a centralised Monitoring and Management centre is required to allow the appropriate team to manage, configure and monitor the system from a single vantage point.
