"Do you know how your taxes are being analysed nowadays?"
This is how Estela Lopes explains her research to people who do not have a law degree, and it usually works, because almost nobody knows the answer.
She explains what happens next. If you live in Europe, the form may well have been filled in for you, and the return you submitted was probably assessed by an algorithm rather than read by an official. The software works out the figure at the bottom, the money you owe or the money coming back. And in a great many cases, the tax authority did not build it.
Tax is not actually what Estela researches, but she reaches for it because everybody files a return. What interests her is where the software came from.
The numbers are not small. In her work she cites a European Commission study that counted 940 cases of AI use in the public sector across 30 European countries in 2024, from road traffic enforcement to welfare benefits, crime prediction, and decisions in migration and asylum procedures. At the EU level, she notes, the European Data Protection Supervisor has looked at the systems the institutions themselves use, and found that most did not originate inside them.
"They are either bought from a private actor directly by the public authority, or they are developed in collaboration with that private actor," says Estela, a PhD researcher in the EUI Department of Law. "My research looks into those collaborations."
Why buying AI is not like buying printers
Most people then ask the sensible question: Governments buy a lot of other things from private companies, so why should we care about where the software comes from?
Estela has an answer ready. "They buy printers, they buy computers," she says. "And in the majority of cases that is not an issue, because you are just buying equipment that you are going to use in your activity." An AI system, in her account, belongs in a whole different category. It does the job of the public servant, and at the end of it somebody receives a decision about their money, their benefits or their right to cross a border.
"If you have a gateway for a private company to influence an administrative decision that will affect you, the citizen, I think that this can be a problem," she says. "This can be a gateway for private interests to enter into public matters."
She came to her research topic through practice. Before Florence, Estela was a lawyer in Lisbon, working on public contracts for software and early algorithms, mostly advising the private side and occasionally the public one, first on winning tenders and then on performing the contracts. She was teaching at the same time, at the Catholic University of Portugal, and that is what eventually decided things. "My heart was more towards academia," she says. "I loved teaching, and I still do."
But the research question came from practice. "It came from seeing the imbalances in the relationship," she says, "the power that the private party was increasingly having in relation to the public authority."
Inside eu-LISA, the agency that builds EU border systems
Her first case study is an agency almost nobody could name, which is rather the point. eu-LISA runs the large-scale IT systems behind the EU's Area of Freedom, Security and Justice, building the tools that automate visa processing and travel authorisations. Estela notes it works largely through external contractors, who are embedded in its daily operation. She is upfront that she had never heard of it either until she started looking.
The system she follows is called ETIAS, still under construction. When it goes live, it will decide whether visa-exempt travellers from outside the EU may enter the Schengen area. Someone flying from Canada to Portugal, for instance, will apply through it.
She chose it because she wanted a case people might feel directly affected by rather than watch from the sidelines. She knows no EU citizen will ever apply to ETIAS. But anyone flying from Europe to the United States meets the same thing in its American counterpart, ESTA. In both cases, an algorithm screens each application for security and public health risks before anyone boards.
"Even though you are not affected by this specific system, you are affected by other similar ones," she says.
Nobody yet knows how ETIAS will behave, because it is not switched on. What Estela can already describe is who is making the choices that will determine how it behaves, and in her account it is a mixture of the agency's own staff and the contractors working alongside them.
"The contractors are the ones that in their daily work are in some respects deciding the design choices of the system, its thresholds and weights," she says.
Two things worry her in this scenario. The first is something she can already see in the paperwork: Agencies that hire private firms to build a system tend to hire private firms to maintain it, and they are often the same firms. "You become dependent on private companies to go about your daily activity," she says.
The second she cannot prove, and she says so before anyone asks. She suspects that private values may end up baked into the way these systems decide. "This is my intuition, but it is extremely difficult to prove," she says. "And this is a question for me as I am doing my research, but also for us as a society."
Can citizens have a say in public sector AI?
There is a popular answer to all of this, which is to let citizens in. Give people a say in whether a system gets built and how it is designed, the argument runs, and the values inside it are more likely to be public ones. Estela spent her first year at the EUI testing that idea, in a paper recently published in the Cambridge Forum on AI: Law and Governance. She went looking for the way in, through constitutional and administrative law, through public procurement, through the AI Act, and she did not find one.
There is no way in.
The AI Act does make room for participation, she says, but not where it counts. You can have a say in drawing up standards and codes, or sit on the bodies that oversee the Act. What you cannot do is have a say in the system itself. And the room it makes is for stakeholders, meaning people with an interest or some expertise, rather than the person who simply ends up on the receiving end.
One right does go further. Anyone can ask an EU institution to hand over its documents, and by Estela's reading that ought to include the paperwork behind whatever AI it is building. Except that the rules also protect commercial interests, she notes, and a contractor's proprietary information is exactly that. As she puts it, the firm that built the system is in many cases the reason you cannot see how and what it built.
None of this is only about the EU level. Estela has no idea what, for instance, the Portuguese or Italian governments are building right now, and neither, she points out, does anybody else. "The way governments are developing new systems, new technologies, to apply in public decision-making is still very far away from its citizens," she says. "And there is hardly any way of knowing it."
What fieldwork reveals that contracts do not
Finding out what actually happens has meant getting out of the library. Estela files requests to see documents, interviews officials inside the institutions she studies and the companies working with them. In early September, she sat in on a roundtable that eu-LISA holds with its private contractors.
The fieldwork is not finished, so she talks about the pattern rather than the particulars. What she has found is that public bodies and private firms work together far more closely than the contracts would suggest.
"The only way for you to really understand how the contracts are being performed is by talking with the people that are involved in the performance of these contracts," she says.
Nobody really taught her to do that. Portuguese legal education is doctrinal, she explains, concerned with interpreting what the law says. "I do not want to call doctrinal research old-fashioned, because it is a very important part of what lawyers do. It is our core." What it leaves out, in her view, is what people are actually doing, and how the law is being shaped in practice. The EUI Department of Law, she says, was open enough to let her go and find out. "I believe the EUI is a spectacular place for that."
That same move, from the text to the practice, is where her argument ends up. Administrative lawyers, in her account, usually look at the decision after it has been made, asking whether the right procedure was followed and whether reasons were given. Where AI is involved in administrative decision-making, Estela thinks the decisive moment happens earlier.
"More and more, we need to shift from that post-decision moment to the pre-decision moment," she says. "How was the system that made that decision developed? Did its development follow all the procedural safeguards that it needed to follow? As lawyers, we also need to update our toolkit."
She is not campaigning against any of it. The technology makes public administration faster and easier to deal with, she says, and she believes in that. She also takes it as given that the partnerships will keep growing, because in her view no public authority is going to build these systems from scratch, and certainly not a generative model.
"We will see more and more private companies involved in public activity as we increasingly digitalise the public sector," she says.
What she keeps coming back to is where the boundary shall sit, and she thinks about it as a researcher and as someone who files a tax return like everyone else. Is there still a line between the public sector and the private one, and if so, where does it fall?
"I think we as a society need to think about what the limit is. Are we comfortable with these collaborations, or do we want to draw more boundaries?"
Estela Lopes is a PhD researcher in the EUI Department of Law. Her thesis, “Outsourcing Public Intelligence: Rethinking Delegation in Public-Sector Artificial Intelligence Development Outsourcing”, is supervised by Deirdre Curtin.