As new technologies like low-code/no-code development and generative AI (GenAI) revolutionize how we build and interact with software, they also bring about new security challenges—especially for the public sector. Protecting sensitive information and online accounts is more critical than ever, as cybercriminals look to exploit gaps in these emerging systems. Ensuring robust security and threat visibility is now essential for safeguarding against the risks associated with these advancements, especially as traditional safeguards become less effective in the face of evolving threats.
Low-code Development Exposes New Risk
One of the unintended consequences of our shift to a low-code/no-code development paradigm is the delegation of complex development tasks to Large Language Models (LLMs) and GenAI systems, often bypassing seasoned developers and architects. This opens new opportunities for cybercriminals. These systems excel at functional requirements—‘Build me a website that accepts customer checkout requests’—but they rarely infer non-functional needs, like security, unless explicitly instructed.
In traditional software development, security considerations are often implicit, stemming from the experience of developers and architects who’ve spent years learning from real-world failures. GenAI, however, lacks this depth of experience and focuses narrowly on the task at hand. The result? Incomplete or inadequate security measures in software developed through these systems. As organizations lean more heavily on GenAI, we risk creating an insecure software ecosystem ripe for exploitation by threat actors.
The Proliferation of Knowledge-Based Verification Attacks
We’re on the brink of a surge in automated attacks exploiting vulnerabilities in Knowledge-Based Verification (KBV) systems. Large-scale data breaches, like the one that exposed millions of Social Security numbers last year, are eroding the effectiveness of this approach at confirming identity when creating an account or supporting a password reset. These processes often rely on KBV—such as answering questions about your mother’s maiden name or the street you grew up on—but this information is increasingly accessible to malicious actors.
As these personal details become more widely available through data breaches and online marketplaces, attackers can easily bypass KBV systems. Worse yet, threat actors can now leverage LLMs to develop sophisticated tools to mine personal data at scale and orchestrate automated attacks against these KBV systems. Organizations face an urgent challenge: how to protect accounts in a world where traditional KBV methods are no longer secure or reliable while still offering users a legitimate path to create an account or regain access when needed.
LLM Safeguards Can Be Overridden or Bypassed by Running Models Locally
With the proliferation of local LLM instances and tools like Ollama, we’ll see safeguards embedded in commercial LLMs eroded or bypassed entirely. Running models locally can allow threat actors to fine-tune them, removing restrictions on malicious activity and enabling custom models optimized for cybercrime. This creates a new frontier for scaled attacks that are faster, more targeted, and harder to detect until it’s too late.
Imagine a threat actor fine-tuning a model to craft phishing campaigns, identify vulnerabilities in software, or automate account takeovers. The ability to localize and modify these models fundamentally shifts the balance, empowering attackers with tools tailored to their malicious intent. The guardrails built into commercial LLMs are no match for this growing trend, amplifying the need for robust detection and defense strategies at every level.
As the public sector continues to adopt innovative technologies, staying ahead of emerging cyber threats is crucial. The increasing sophistication of attacks, such as those targeting KBV systems and leveraging GenAI, highlights the need for stronger protections. By prioritizing comprehensive security measures and threat detection, organizations can mitigate the risks of these evolving vulnerabilities and safeguard their sensitive data and online accounts against malicious actors. It is essential to build and maintain resilient security strategies to ensure the integrity of digital infrastructures in this rapidly changing environment.
To learn more about how HUMAN Security helps the public sector protect citizen accounts, sensitive information, and critical infrastructure, click here.
Carahsoft Technology Corp. is The Trusted Government IT Solutions Provider, supporting Public Sector organizations across Federal, State and Local Government agencies and Education and Healthcare markets. As the Master Government Aggregator for our vendor partners, including HUMAN Security, we deliver solutions for Geospatial, Cybersecurity, MultiCloud, DevSecOps, Artificial Intelligence, Customer Experience and Engagement, Open Source and more. Working with resellers, systems integrators and consultants, our sales and marketing teams provide industry leading IT products, services and training through hundreds of contract vehicles. Explore the Carahsoft Blog to learn more about the latest trends in Government technology markets and solutions, as well as Carahsoft’s ecosystem of partner thought-leaders.
Generative AI is still fairly new to the education space and educators are on both sides of the spectrum of acceptance—some prefer to erase it from their schools while others are open to embracing the up-and-coming technology for use cases not only in the classroom, but also to prepare students for the future workforce.
Last year set a landmark standard for innovation in artificial intelligence (AI). Federal, State, and Local Governments and Federal Systems Integrators are eager to learn how they can implement AI technology within their agencies. With the recent Presidential Executive Order for AI, many Public Sector-focused events in 2024 will explore AI modernizations, from accelerated computing in cloud to the data center, secure generative AI, cybersecurity, workforce planning and more.
USSF Supra Coders Utilize DevSecOps for Innovation
The primary focus of AI is to use data and computations to aid in decision-making. Generative AI can create text responses, videos, images, code, 3D products and more. AI as a Service, cloud-based offerings of AI, helps experts get work done more efficiently by advancing infrastructure at a quicker pace. In contrast, AI is also commonly used by the general public as a toy, since its responses can sometimes be entertaining. The comfort users have with AI and wide range of inputs introduces risk, and these risks can proliferate exponentially.
In all learning environments, students and teachers rely on modern technologies to enhance their experiences to be as informational, productive and efficient as possible. In recent years, hybrid learning and collaborative digital spaces became essential components of education for both K-12 and higher education organizations. With this development, education technology has evolved and expanded to include new and more advanced AI systems inside and outside the classroom.
Innovative trends in education technology have made it possible for students and staff to stay connected, whether through remote online learning or collaborative learning in the classroom. AI tools like 
For teachers, implementing AI can help automate repetitive daily tasks like grading tests and quizzes, and catching minor mistakes within written essays. This way they have more freedom and time to focus on in-depth feedback, creating comprehensive lesson plans and spending one-on-one time with their students. Additionally, AI tools can give students instant feedback on their work, allowing them to be more independent in identifying inaccuracies and recognizing successful projects.[
Augmented reality (AR) and virtual reality (VR) are becoming increasingly more popular in students’ everyday lives, so using these technologies as a learning tool is familiar and compelling for them to gain valuable experiences in the classroom. Immersive technologies can simulate real-world scenarios for students to gain hands-on experience with low risk, like medical simulations and technical experiments. It also can allow students to break the barrier between their physical space and complex concepts like observing the planets up close or enlarging and examining something microscopic.[
Throughout a student’s education, data is continually collected to better understand and predict their developing needs and most effective learning strategies. AI technologies can quickly and automatically analyze and report on this data, allowing teachers and professors to evaluate trends in an individual student’s or an entire class’s performance. Empowered with this knowledge, educators can tailor their lesson plans and take a more proactive approach to supporting students’ needs, ultimately increasing academic improvement for all.[
Student’s learning styles can vary depending on many factors. For example, some students learn best through more visual and interactive experiences, while others may learn best through memorization and flashcards. Analyzing data collected by AI can help teachers be more informed and prepared educators for different kinds of learners. By applying the insights gathered from AI algorithms, educators can create personalized tracks for individual students, including aspects like adjusting the types of content, working with their comfortability, tailoring to their pace of learning and understanding their comprehension of learning objectives.[