1. médialab Sciences Po
  2. Activities
  3. Écologies des pratiques LLMs

Écologies des pratiques LLMs

How can we reframe the role of LLMs in ordinary work practices?

The Ecologies of LLM Practices (EL2MP) project creates a space of inquiry to document the role of LLMs in various professional practice(s) and the consequences arising from their use.

Research

Context

The booming rise of large language models (LLMs) such as ChatGPT has sparked a rush to produce discourse about these technologies. The quick crystallisation of a shared outlook around a few key themes has narrowed the scope of potential interrogations. Public and scientific debates focus on technical issues: algorithmic bias, confabulation, and intellectual property violations. However, the problems and consequences associated with their actual use – for both their users and their professional contexts – remain largely unexplored. This asymmetry fuels a mechanical view of technological development and its effects, as if the technical analysis of these systems were enough to predict their social impact. Moreover, these discourses present AI as a monolithic and disruptive entity, dismissing the possibility that it may be aligned with existing practices and that its effects may vary depending on situations encountered in one’s job.

There is thus an urgent need to move beyond predictions about the future of work, to account for the professional contexts in which LLMs are used, and to identify current issues – not prospective ones. How do AI's well-known problems (bias, confabulation, etc.) manifest in established practices? What new, unexpected problems are surfacing? How do LLMs shape individual work practices? And, in turn, how do professional environments shape LLMs and their use?

Objective

To answer these questions, the Ecologies of LLM Practices (EL2MP) project creates research areas dedicated to workers for them to document and reflect on their use of LLMs. Our investigation aims to highlight the “savoir-faire”, expertise, and values of workers rather than those of AI designers or economic decision-makers. The project aims to scrupulously examine how LLMs fit into various professional practices. EL2MP will investigate how users relate to LLMs in terms of:

  • Evaluation: How do professionals assess the value LLMs add or remove?
  • Effort: What new kinds of work do LLMs require from their users?
  • Perception: How do workers evaluate and perceive LLMs over time as they continue working with them?

Methodology

To carry out the project, we developed an experimental research protocol based on the active involvement of participants. The fruit of a collaboration between sociologists, science and technology researchers (STS), and designers, this protocol aims to create:

  • Room for hesitation: Through a series of exercises, the protocol provides participants with various means of documenting and reflecting on their use of LLMs. We aim to establish a framework where doubt and hesitation are not only accepted but encouraged. Some exercises introduce deliberate pauses in professional routines, causing participants to take a step back from their own practices: during these pauses, judgment is suspended, giving users enough room to question their instinctive habits and feel unsure about what they once thought was certain. Other exercises aim to intensify the use of LLMs to shed light on what would otherwise remain too subtle to be perceived.
  • An ecological archive of practices: The digital traces from LLM use serve as the starting point for the exercises we designed. They are analysed, contextualised, and discussed individually and collectively. Through this process, participants progressively build a multimodal archive (audio, photos, videos, drawings, and logbooks) that tangibly reflects their LLMs experience and how they are incorporated into various professional norms and material configurations.
  • A workbook (vademecum): The workbook collects all the exercises and their instructions, organized into thematic blocks. Designed as a modular object, where pages can be added or removed as needed, it accompanies each participant throughout the protocol. The vademecum serves as both the material support for the protocol and the project’s means of dissemination, as it can be reproduced and adapted in various academic and professional contexts.

In practice, the protocol follows several cohorts, each of which comprising professionals from the same field over a period of 6 to 8 months, totaling 40 co-inquirers. The project relies on Sciences Po’s Master’s Degree and Executive Education programs to recruit highly qualified professionals. The choice of this audience reflects the way technology companies present LLMs, promoting them as a tool (and sometimes a substitute) for this type of activity and level of expertise. The protocol is designed so that it can be implemented in and adapted to other professional sectors in order to provide a broader panorama of LLMs in the working world.

Each cohort carries out the exercises at the same pace, meeting up on a regular basis (between 10 and 15 times over the period) both in workshops led by researchers and in individual interviews. This approach allows researchers to track the evolution of their LLM usage and to distinguish fleeting practices from those that withstand the test of time.