Brunei, as Brunei Darussalam, competed at the 2004 Summer Olympics in Athens, Greece, which took place between 13 and 29 August 2004. The country's participation in Athens marked its fourth appearance at the Summer Olympics since its debut in the 1988 Summer Olympics.

The Brunei delegation included only one track and field athlete, meaning Brunei, along with British Virgin Islands and Liechtenstein, sent the lowest number of athletes to the 2004 Summer Games at one. The athlete selected was Jimmy Anak Ahar, who was a middle-distance runner that was selected via a wildcard as the nation had no athletes that met the "A" or "B" standards in any event. Ahar was also selected as flag bearer for the opening ceremony. Ultimately, Ahar did not progress beyond the heats, meaning Brunei won no medals at this Summer Olympics.

Although Brunei first participated in the Summer Olympic in the 1988 Summer Olympics in Seoul, South Korea, it was only represented by one official. It would not be until the 1996 Summer Olympics in Atlanta, United States that the country would send athletes to the Games. Since then, it has participated in four Summer Olympics between its debut and the 2004 Summer Olympics. No Brunei athlete had ever won a medal at the Summer Olympics before the 2004 Athens Games.

The Brunei National Olympic Committee (NOC) selected one athlete via a wildcard. Usually, a NOC would be able to enter up to three qualified athletes in each individual event as long as each athlete met the "A" standard, or one athlete per event if they met the "B" standard. However, since Brunei had no athletes that met either standard, they were allowed to select an athlete as a wildcard. The one athlete that was selected to compete in the Athens Games was Jimmy Anak Ahar in the Men's 1500 meters. Sending only one athlete to the Athens Games meant that the country, along with British Virgin Islands and Liechtenstein, was notable for sending the lowest number of athletes to the 2004 Summer Games. Ahar was flag bearer for the opening ceremony. Among officials, sports official Sofian Ibrahim represented the country.

Making his Summer Olympic debut, Jimmy Anak Ahar was notable for becoming the youngest ever competitor to represent Brunei at the Olympics aged 22. The age record stood until the 2012 Summer Olympics, when Anderson Chee Wei Lim surpassed it. Ahar qualified for the 2004 Athens Games as a wildcard, as his best time, three minutes, 59.81 seconds at the 2003 Southeast Asian Games Men's 1500 meters, was 21.81 seconds slower than the "B" qualifying standard required. He competed on 20 August in the Men's 1500 meters against thirteen other athletes in the third heat. He ran a time of 4 minutes, 14.11 seconds, finishing 13th. He ranked ahead of Tanzania's Samwel Mwera (who did not start), and behind Guam's Neil Weare (4 minutes, 5.86 seconds), in a heat led by Great Britain's Michael East (3 minutes, 37.37 seconds). Overall, Ahar placed 39th out of the 41 athletes that competed, and was 32.97 seconds behind the slowest athlete that progressed to the next round. Therefore, that was the end of his competition.

Men

Athlete	Event	Heat		Semifinal		Final
		Result	Rank	Result	Rank	Result	Rank
Jimmy Anak Ahar	1500 m	4:14.11	13	did not advance