Staff Software Engineer at Cybersecurity Company Bishop Fox
In the field of cybersecurity, keeping pace with evolving threats is essential. As a Staff Software Engineer at Bishop Fox, I worked on a project to help clients identify vulnerabilities in their digital assets more effectively.
The goal was to develop a system for processing and discovering global asset data that improved performance, scalability, and cost-efficiency in Attack Surface Management (ASM). Here's an overview of the project and my contributions.
The Challenge: Addressing Real-Time Needs in Asset Monitoring
Attack surfaces encompass a company's internet-facing assets, which generate large, dynamic, and complex datasets. Clients required a shift from periodic scans to continuous monitoring.
Key technical challenges included:
- Latency: Traditional methods were not fast enough for transient cloud assets.
- Data Integration: Combining various data sources without performance issues.
- Cost Management: Scaling infrastructure while controlling expenses, especially for secret management and database operations.
The objective was to create a system that was efficient, reliable, and cost-effective.
My Approach: Designing a Scalable Architecture
I adopted a microservices architecture to handle these issues, using Golang for its concurrency features and Kafka for event streaming. This enabled an event-driven system that could scale as needed.
Developing the Discovery Engine
The core component was an engine that integrated public data with Bishop Fox's proprietary methods. By separating discovery logic from data processing, the system supported horizontal scaling for enterprise-level demands.
Building APIs for Integration
I developed RESTful APIs in Golang to facilitate high-throughput communication between services, ensuring type safety and performance.
Key Contributions and Improvements
Throughout the project, I focused on practical enhancements to meet team standards. Here are some of the main achievements:
-
Performance Enhancements I reviewed and optimized our PostgreSQL database queries with better indexing, resulting in a 200% improvement in data retrieval speeds. This allowed quicker access to threat intelligence for analysts and clients.
-
Cost Reductions in Data Management I transitioned sensitive data handling from AWS Secrets Manager to a PostgreSQL-based solution secured by AWS Key Management Service (KMS).
- Outcome: This reduced annual costs from tens of thousands of dollars to nearly zero, highlighting the value of efficient resource use.
-
Improving Code Reliability I implemented an automated testing framework with unit and integration tests, increasing code coverage and promoting a test-driven development approach.
Results and Impact
The system's design led to measurable improvements:
- 700% Faster Discovery: Reduced asset discovery time significantly, helping close potential vulnerability windows.
- Cost Savings on Secrets: Eliminated substantial overhead through integrated PostgreSQL and KMS usage.
- 2x Data Retrieval Efficiency: Improved the data layer for faster threat visualization.
- Better Security Insights: Enabled clients to detect shadow IT and threats via continuous monitoring and proprietary techniques.
Lessons Learned: Insights from Staff Engineering
This project emphasized the importance of combining technical skills with clear documentation. I contributed to our internal knowledge base by documenting the architecture, coding standards, and best practices, which helped share knowledge across the team.
Working in an Agile/Scrum environment, I collaborated with cross-functional teams to translate security needs into functional code.
Conclusion
Developing this platform for global attack surface monitoring required attention to performance and efficiency. Using Golang, AWS, and microservices, we created a solution that's faster, more cost-effective, and scalable.
If you're working on high-performance systems or cloud optimization, feel free to connect on LinkedIn or reach out to discuss event-driven architectures.