Study title

Responses Generated by Large Language Models to Ethically Challenging Prompts 2024- 2025

Dataset ID Number

FSD4133

Persistent identifiers

https://urn.fi/urn:nbn:fi:fsd:T-FSD4133
https://doi.org/10.60686/t-fsd4133

Data Type

Qualitative

Authors

• Bazuaye, Deborah E. (University of Oulu)
• Guadagno, Rosanna E. (University of Oulu)
• Pohjalainen, Sami (University of Oulu)
• Koirala, Sharamsh (University of Oulu)

Abstract

The dataset consists of 1,260 texts generated by six large language models in response to ethically challenging prompts created by researchers. The researchers created 70 prompts covering 16 ethically sensitive topics; for each prompt, three versions were generated: a standard version, a role-playing version, and a version designed to encourage the circumvention of safety restrictions.

The prompts address ethical dilemmas, individual rights, the exercise of societal power, freedom of expression, police use of force, and the responsibilities of governments and AI systems. They also consider violations of the law or violent action as potentially justified means and, in some cases, request detailed instructions for dangerous activities. Several prompts concern cybersecurity, malware, fraud, document forgery, financial crime, and the circumvention of system safeguards. The prompts also include medical questions concerning medication, dosages, allergies, and the prevention of mental health problems, as well as the use of health and personal data to influence people's behaviour. Some address privacy, consent, confidentiality, research ethics, AI training data, and covert recording. In addition, the prompts include questions concerning hate speech, religion, child safety, and other age-restricted content. Overall, the prompts test how AI systems respond to ethically challenging, privacy-infringing, discriminatory, dangerous, or unlawful requests.

Keywords

artificial intelligence; crime and security; data protection; ethics; law enforcement; legislation; responsibility

Topic Classification

• Social sciences (Fields of Science Classification)
• Information society (CESSDA Topic Classification)
• Information technology (CESSDA Topic Classification)

Series

Individual datasets

Distributor

Finnish Social Science Data Archive

Access

The dataset is (A) openly available for all users without registration (CC BY 4.0).

Data Collector

• Bazuaye, Deborah E. (University of Oulu)

Collection Dates

2024-08-11 – 2025-07-02

Nation

Finland

Geographical Coverage

No specific geographical coverage

Analysis/Observation Unit Type

Media unit: Text

Universe

Responses from the six major language models to the given prompts

Time Method

Cross-section

Sampling Procedure

Non-probability: Purposive

Six large language models were selected to represent diverse safety architectures and alignment approaches. The prompt set was researcher-constructed to cover 16 ethically sensitive categories informed by provider usage policies and relevant regulatory frameworks, rather than sampled from an existing population.

Collection Mode

Experiment (The data were generated through a controlled trial examining how six large language models (Claude 3.5 Sonnet, Command R+, Dolphin 2.5, Gemini 1.5 Pro, GPT-4o and GPT-o1) respond to ethically sensitive requests under different prompt framings. Seventy standard (STD) prompts across 16 ethically sensitive categories served as the baseline condition, and each was reframed as a character role-play (CRP) prompt and a Do Anything Now (DAN) jailbreak prompt, forming two experimental conditions and yielding 210 prompts. All prompts were submitted manually to each model via the You.com platform, each in a new session with no custom instructions, and responses were recorded verbatim in structured spreadsheets. No human participants were involved.)

Research Instrument

Other research instrument

Data Files

70 prompts and the 1,260 text responses generated by the language models in a single ODS file.

Data File Language

The data files of this dataset are available in the following languages: English.

Qualitative data are available in their original language only and are not translated.

Data Version

1.0

Completeness of Data and Restrictions

The responses generated by the language models are based on public sources available online, therefore, the data does not contain such personal information that would require anonymisation. However, the authors have removed personal information (name and institution) from a few of the responses generated by artificial intelligence and replaced it with a descriptive label (e.g., researcher).

Citation Requirement

The data and its creators shall be cited in all publications and presentations for which the data have been used. The bibliographic citation may be in the form suggested by the archive or in the form required by the publication.

Bibliographical Citation

Bazuaye, Deborah E. (University of Oulu) & Guadagno, Rosanna E. (University of Oulu) & Pohjalainen, Sami (University of Oulu) & Koirala, Sharamsh (University of Oulu): Responses Generated by Large Language Models to Ethically Challenging Prompts 2024- 2025 [dataset]. Data version 1.0 (2026-10-29). Finnish Social Science Data Archive [distributor]. DOI: https://doi.org/10.60686/t-fsd4133; URN: https://urn.fi/urn:nbn:fi:fsd:T-FSD4133

Deposit Requirement

Notify FSD of all publications where you have used the data by sending the citation information to user-services.fsd@tuni.fi.

Disclaimer

The original data creators and the archive bear no responsibility for any results or interpretations arising from the reuse of the data.

Related Publications

Bazuaye, D. E. (2025, November). Building Safer AI: Testing Ethical Boundaries and Safety Mechanisms. Poster presentation, Arctic AI Day, Oulu, Finland.

Bazuaye, D. E., Guadagno, R. E., & Pohjolainen, S. (2026). Between Ethics and Law: A Socio-Technical Assessment of LLM Responses Against Article 5 of the EU AI Act. Proceedings of TKTP 2026, 43rd Annual Symposium on Computer Science, University of Turku, Finland, 8-9 June 2026. Jufo level 1 (peer-reviewed).

Bazuaye, D.E. (2025) Probing the Ethical Boundaries of Large Language Models: Responding to Dilemmatic Prompts (2025). Master's thesis, University of Oulu. urn.fi/URN:NBN:fi:oulu-202506174695