How to get data as a stream from a WebAPI (.NET)
A remote web API can be consumed as a stream, which can take longer, depends an other server process, and the necessary time to generate the requested content.
It is a real scenario, if we get event, data pieces from a services, etc.
In this project I will demonstrate, how we can use a streaming API, how we can iterate through on the incoming dataset with the IAsyncEnumerable interface.
This controller generates a set of weather data, than send the dataset back to the client as a stream. Every data item serialized into Json format, and finally it sends '[DONE]' string to indicate, the streaming is over.
[ApiController] [Route("[controller]")] public class WeatherForecastController : ControllerBase { private static readonly string[] Summaries = new[] { "Freezing", "Bracing", "Chilly", "Cool", "Mild", "Warm", "Balmy", "Hot", "Sweltering", "Scorching" }; private readonly ILogger<WeatherForecastController> _logger; public WeatherForecastController(ILogger<WeatherForecastController> logger) { _logger = logger; } [HttpGet] public async Task GetAsync() { Response.StatusCode = 200; Response.ContentType = "text/html"; List<WeatherForecast> list = Enumerable.Range(1, 5) .Select(index => new WeatherForecast { Date = DateTime.Now.AddDays(index), TemperatureC = Random.Shared.Next(-20, 55), Summary = Summaries[Random.Shared.Next(Summaries.Length)] }) .ToList(); StreamWriter sw; await using ((sw = new StreamWriter(Response.Body)) .ConfigureAwait(false)) { foreach (WeatherForecast item in list) { // Thread.Sleep simulates a long running process, // which generates some kind of output Thread.Sleep(1000); await sw.WriteLineAsync(item.ToString()).ConfigureAwait(false); await sw.FlushAsync().ConfigureAwait(false); } await sw.WriteLineAsync("[DONE]").ConfigureAwait(false); }; } }On the client side, we need a caller code, which can read and handle the incoming data. Check out the 'yield' instruction when passing every incoming data item towards the API caller. If the code detects the '[DONE]' maker, the reading ends.
public class Api { public async IAsyncEnumerable<string> StreamedAsync([System.Runtime.CompilerServices.EnumeratorCancellation] CancellationToken cancellationToken = default) { using (HttpRequestMessage request = new HttpRequestMessage(HttpMethod.Get, "https://localhost:7176/weatherforecast")) { using (HttpClient httpClient = new HttpClient()) { //httpClient.Timeout = Timeout.InfiniteTimeSpan; using (HttpResponseMessage response = await httpClient.SendAsync(request, HttpCompletionOption.ResponseHeadersRead, cancellationToken).ConfigureAwait(false)) { if (response.IsSuccessStatusCode) { using (Stream contentStream = await response.Content.ReadAsStreamAsync().ConfigureAwait(false)) { using (StreamReader reader = new StreamReader(contentStream)) { string? line = null; while ((line = await reader.ReadLineAsync().ConfigureAwait(false)) != null && !cancellationToken.IsCancellationRequested) { if ("[DONE]".Equals(line.Trim())) break; if (!string.IsNullOrWhiteSpace(line)) { yield return line; } } } } } else { string? jsonResult = await response.Content.ReadAsStringAsync().ConfigureAwait(false); yield return jsonResult; } } } } } }Finally, a code can use the Api class to acquire the data, using IAsyncEnumerable on a proper way.
This code iterates on the incoming data asynchronously and display the given data as a text, one by one, after a data item arrived. This approch does not wait to arrive all of the data, it immediatelly displays what we got from the server.
static async Task Main(string[] args) { Console.WriteLine("Press a key to start"); Console.ReadKey(); Api api = new Api(); await foreach (string line in api.StreamedAsync(CancellationToken.None)) { Console.WriteLine(line); } Console.WriteLine("[DONE] press a key to exit"); Console.ReadKey(); }