Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add MEAI.Evaluation library #10818

Open
wants to merge 8 commits into
base: main
Choose a base branch
from
Open

Add MEAI.Evaluation library #10818

wants to merge 8 commits into from

Conversation

 Build.Reason:Manual by Genevieve Warren
 Build.Url:https://apidrop.visualstudio.com/Content%20CI/_build/results?buildId=488911&view=results
 source_repo.branch:meai-evaluation
 source_repo.url:https://apidrop.visualstudio.com/_git/binaries
 Build.Reason:Manual by Genevieve Warren
 Build.Url:https://apidrop.visualstudio.com/Content%20CI/_build/results?buildId=488969&view=results
 source_repo.branch:meai-evaluation
 source_repo.url:https://apidrop.visualstudio.com/_git/binaries
Copy link
Contributor

@gewarren - This PR edits one or more files whose 'source of truth' for documentation is not in this repo. Please make documentation updates in the /// comments in the dotnet/runtime repo (or dotnet/extensions repo) instead.

This comment was marked as outdated.

 Build.Reason:Manual by Genevieve Warren
 Build.Url:https://apidrop.visualstudio.com/Content%20CI/_build/results?buildId=489437&view=results
 source_repo.branch:meai-evaluation
 source_repo.url:https://apidrop.visualstudio.com/_git/binaries

This comment was marked as outdated.

@gewarren gewarren changed the title Add MEAI.Evaluation package Add MEAI.Evaluation library Jan 9, 2025
@gewarren gewarren marked this pull request as ready for review January 9, 2025 04:48

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

Copy link

Learn Build status updates of commit 8b223f2:

⚠️ Validation status: warnings

File Status Preview URL Details
xml/Microsoft.Extensions.AI.Evaluation.Quality/ChatConversationEvaluator.xml ⚠️Warning View Details
xml/Microsoft.Extensions.AI.Evaluation.Quality/SingleNumericMetricEvaluator.xml ⚠️Warning View Details
xml/Microsoft.Extensions.AI/ChatClientBuilder.xml ⚠️Warning View Details
xml/Microsoft.Extensions.AI/ChatClientBuilderServiceCollectionExtensions.xml ⚠️Warning n/a (file deleted or renamed) Details
xml/Microsoft.Extensions.AI/ConfigureOptionsChatClientBuilderExtensions.xml ⚠️Warning View Details
xml/Microsoft.Extensions.AI/ConfigureOptionsEmbeddingGeneratorBuilderExtensions.xml ⚠️Warning View Details
xml/Microsoft.Extensions.AI/EmbeddingGeneratorBuilder`2.xml ⚠️Warning View Details
xml/Microsoft.Extensions.AI/EmbeddingGeneratorBuilderServiceCollectionExtensions.xml ⚠️Warning n/a (file deleted or renamed) Details
xml/Microsoft.Extensions.AI/OpenAIRealtimeExtensions.xml ⚠️Warning View Details
xml/System.Runtime.Serialization/Formatter.xml ⚠️Warning View Details
xml/System.ServiceModel.Channels/HttpRequestMessageExtensionMethods.xml ⚠️Warning View Details
xml/System.ServiceModel.Channels/HttpRequestMessageProperty.xml ⚠️Warning View Details
xml/System.ServiceModel.Channels/HttpResponseMessageProperty.xml ⚠️Warning View Details
[xml/System.ServiceModel.Channels/NetFramingTransportChannelFactory1.xml](https://github.com/dotnet/dotnet-api-docs/blob/meai-evaluation/xml/System.ServiceModel.Channels/NetFramingTransportChannelFactory1.xml) ⚠️Warning View Details
xml/System.ServiceModel.Federation/IWSTrustChannelContract.xml ⚠️Warning View Details
xml/System.ServiceModel.Federation/WSTrustChannel.xml ⚠️Warning View Details
xml/System.ServiceModel.Federation/WSTrustChannelSecurityTokenProvider.xml ⚠️Warning View Details
xml/System.ServiceModel.Federation/WSTrustTokenParameters.xml ⚠️Warning View Details
xml/System.ServiceModel/HttpTransportSecurity.xml ⚠️Warning View Details
xml/Microsoft.Extensions.AI/AdditionalPropertiesDictionary.xml 💡Suggestion View Details
xml/Microsoft.Extensions.AI/AnonymousDelegatingChatClient.xml 💡Suggestion View Details
xml/Microsoft.Extensions.AI/AnonymousDelegatingEmbeddingGenerator`2.xml 💡Suggestion View Details
xml/Microsoft.Extensions.AI/CachingChatClient.xml 💡Suggestion View Details
xml/Microsoft.Extensions.AI/CachingEmbeddingGenerator`2.xml 💡Suggestion View Details
xml/Microsoft.Extensions.AI/ConfigureOptionsChatClient.xml 💡Suggestion View Details

This comment lists only the first 25 files in the pull request.

xml/Microsoft.Extensions.AI.Evaluation.Quality/ChatConversationEvaluator.xml

  • Line 0, Column 0: [Warning: xref-not-found - See documentation] Cross reference not found: 'result'.

xml/Microsoft.Extensions.AI.Evaluation.Quality/SingleNumericMetricEvaluator.xml

  • Line 0, Column 0: [Warning: xref-not-found - See documentation] Cross reference not found: 'result'.

xml/Microsoft.Extensions.AI/ChatClientBuilder.xml

  • [Warning: file-not-redirected - See documentation] File xml/Microsoft.Extensions.AI/ChatClientBuilder.xml with URL /dotnet/api/microsoft.extensions.ai.chatclientbuilder.services was deleted without redirection. To avoid broken links, add a redirection.

xml/Microsoft.Extensions.AI/ChatClientBuilderServiceCollectionExtensions.xml

  • [Warning: file-not-redirected - See documentation] File xml/Microsoft.Extensions.AI/ChatClientBuilderServiceCollectionExtensions.xml with URL /dotnet/api/microsoft.extensions.ai.chatclientbuilderservicecollectionextensions was deleted without redirection. To avoid broken links, add a redirection.
  • [Warning: file-not-redirected - See documentation] File xml/Microsoft.Extensions.AI/ChatClientBuilderServiceCollectionExtensions.xml with URL /dotnet/api/microsoft.extensions.ai.chatclientbuilderservicecollectionextensions.addchatclient was deleted without redirection. To avoid broken links, add a redirection.
  • [Warning: file-not-redirected - See documentation] File xml/Microsoft.Extensions.AI/ChatClientBuilderServiceCollectionExtensions.xml with URL /dotnet/api/microsoft.extensions.ai.chatclientbuilderservicecollectionextensions.addkeyedchatclient was deleted without redirection. To avoid broken links, add a redirection.

xml/Microsoft.Extensions.AI/ConfigureOptionsChatClientBuilderExtensions.xml

  • [Warning: file-not-redirected - See documentation] File xml/Microsoft.Extensions.AI/ConfigureOptionsChatClientBuilderExtensions.xml with URL /dotnet/api/microsoft.extensions.ai.configureoptionschatclientbuilderextensions.usechatoptions was deleted without redirection. To avoid broken links, add a redirection.

xml/Microsoft.Extensions.AI/ConfigureOptionsEmbeddingGeneratorBuilderExtensions.xml

  • [Warning: file-not-redirected - See documentation] File xml/Microsoft.Extensions.AI/ConfigureOptionsEmbeddingGeneratorBuilderExtensions.xml with URL /dotnet/api/microsoft.extensions.ai.configureoptionsembeddinggeneratorbuilderextensions.useembeddinggenerationoptions was deleted without redirection. To avoid broken links, add a redirection.

xml/Microsoft.Extensions.AI/EmbeddingGeneratorBuilder`2.xml

  • [Warning: file-not-redirected - See documentation] File xml/Microsoft.Extensions.AI/EmbeddingGeneratorBuilder`2.xml with URL /dotnet/api/microsoft.extensions.ai.embeddinggeneratorbuilder-2.services was deleted without redirection. To avoid broken links, add a redirection.

xml/Microsoft.Extensions.AI/EmbeddingGeneratorBuilderServiceCollectionExtensions.xml

  • [Warning: file-not-redirected - See documentation] File xml/Microsoft.Extensions.AI/EmbeddingGeneratorBuilderServiceCollectionExtensions.xml with URL /dotnet/api/microsoft.extensions.ai.embeddinggeneratorbuilderservicecollectionextensions was deleted without redirection. To avoid broken links, add a redirection.
  • [Warning: file-not-redirected - See documentation] File xml/Microsoft.Extensions.AI/EmbeddingGeneratorBuilderServiceCollectionExtensions.xml with URL /dotnet/api/microsoft.extensions.ai.embeddinggeneratorbuilderservicecollectionextensions.addembeddinggenerator was deleted without redirection. To avoid broken links, add a redirection.
  • [Warning: file-not-redirected - See documentation] File xml/Microsoft.Extensions.AI/EmbeddingGeneratorBuilderServiceCollectionExtensions.xml with URL /dotnet/api/microsoft.extensions.ai.embeddinggeneratorbuilderservicecollectionextensions.addkeyedembeddinggenerator was deleted without redirection. To avoid broken links, add a redirection.

xml/Microsoft.Extensions.AI/OpenAIRealtimeExtensions.xml

  • Line 0, Column 0: [Warning: xref-not-found - See documentation] Cross reference not found: 'OpenAI.RealtimeConversation.ConversationFunctionTool'.
  • Line 0, Column 0: [Warning: xref-not-found - See documentation] Cross reference not found: 'OpenAI.RealtimeConversation.RealtimeConversationClient'.
  • Line 0, Column 0: [Warning: xref-not-found - See documentation] Cross reference not found: 'OpenAI.RealtimeConversation.RealtimeConversationSession'.
  • Line 0, Column 0: [Warning: xref-not-found - See documentation] Cross reference not found: 'OpenAI.RealtimeConversation.ConversationFunctionTool'.
  • Line 0, Column 0: [Warning: xref-not-found - See documentation] Cross reference not found: 'OpenAI.RealtimeConversation.ConversationFunctionTool'.
  • Line 0, Column 0: [Warning: xref-not-found - See documentation] Cross reference not found: 'OpenAI.RealtimeConversation.RealtimeConversationClient'.
  • Line 0, Column 0: [Warning: xref-not-found - See documentation] Cross reference not found: 'OpenAI.RealtimeConversation.RealtimeConversationSession'.
  • Line 0, Column 0: [Warning: xref-not-found - See documentation] Cross reference not found: 'OpenAI.RealtimeConversation.RealtimeConversationSession'.
  • Line 0, Column 0: [Warning: xref-not-found - See documentation] Cross reference not found: 'OpenAI.RealtimeConversation.ConversationUpdate'.
  • Line 0, Column 0: [Warning: xref-not-found - See documentation] Cross reference not found: 'OpenAI.RealtimeConversation.ConversationUpdate'.

xml/System.Runtime.Serialization/Formatter.xml

  • Line 0, Column 0: [Warning: xref-not-found - See documentation] Cross reference not found: 'System.Runtime.Serialization.IFormatter.RequiresDynamicCodeMessage'.
  • Line 0, Column 0: [Warning: xref-not-found - See documentation] Cross reference not found: 'System.Runtime.Serialization.IFormatter.RequiresUnreferencedCodeMessage'.
  • Line 0, Column 0: [Warning: xref-not-found - See documentation] Cross reference not found: 'System.Runtime.Serialization.IFormatter.RequiresDynamicCodeMessage'.

This comment lists only the first 25 errors (including error/warning/suggestion) in the pull request.
For more details, please refer to the build report.

Note: Your PR may contain errors or warnings or suggestions unrelated to the files you changed. This happens when external dependencies like GitHub alias, Microsoft alias, cross repo links are updated. Please use these instructions to resolve them.

For any questions, please:

@gewarren
Copy link
Contributor Author

gewarren commented Jan 9, 2025

@shyamnamboodiripad Can you give it an approval so I can merge it?

@shyamnamboodiripad
Copy link

@shyamnamboodiripad Can you give it an approval so I can merge it?

@gewarren Ah I assumed we would want to update after fixing the warnings above. We just published latest version with the warning fix that also includes a couple of other public API changes and corresponding doc updates -

https://www.nuget.org/packages/Microsoft.Extensions.AI.Evaluation/0.9.56-preview
https://www.nuget.org/packages/Microsoft.Extensions.AI.Evaluation.Reporting/0.9.56-preview
https://www.nuget.org/packages/Microsoft.Extensions.AI.Evaluation.Quality/0.9.56-preview

Would it be too much work to run an update with the new packages? Happy to approve the current PR if you think it may not be worth it...

@gewarren
Copy link
Contributor Author

@shyamnamboodiripad Sure I will rerun with the new version.

 Build.Reason:Manual by Genevieve Warren
 Build.Url:https://apidrop.visualstudio.com/Content%20CI/_build/results?buildId=490009&view=results
 source_repo.branch:meai-evaluation
 source_repo.url:https://apidrop.visualstudio.com/_git/binaries
Copy link

Learn Build status updates of commit 8bcb300:

⚠️ Validation status: warnings

File Status Preview URL Details
xml/Microsoft.Extensions.AI/ChatClientBuilder.xml ⚠️Warning View Details
xml/Microsoft.Extensions.AI/ChatClientBuilderServiceCollectionExtensions.xml ⚠️Warning n/a (file deleted or renamed) Details
xml/Microsoft.Extensions.AI/ConfigureOptionsChatClientBuilderExtensions.xml ⚠️Warning View Details
xml/Microsoft.Extensions.AI/ConfigureOptionsEmbeddingGeneratorBuilderExtensions.xml ⚠️Warning View Details
xml/Microsoft.Extensions.AI/EmbeddingGeneratorBuilder`2.xml ⚠️Warning View Details
xml/Microsoft.Extensions.AI/EmbeddingGeneratorBuilderServiceCollectionExtensions.xml ⚠️Warning n/a (file deleted or renamed) Details
xml/Microsoft.Extensions.AI/OpenAIRealtimeExtensions.xml ⚠️Warning View Details
xml/Microsoft.Extensions.DependencyInjection/OptionsServiceCollectionExtensions.xml ⚠️Warning View Details
xml/Microsoft.Extensions.DependencyInjection/ServiceProviderOptions.xml ⚠️Warning View Details
xml/Microsoft.Extensions.FileProviders/PhysicalFileProvider.xml ⚠️Warning View Details
xml/Microsoft.Extensions.Hosting/IHost.xml ⚠️Warning View Details
[xml/Microsoft.Extensions.Options/OptionsBuilder1.xml](https://github.com/dotnet/dotnet-api-docs/blob/meai-evaluation/xml/Microsoft.Extensions.Options/OptionsBuilder1.xml) ⚠️Warning View Details
xml/System.Numerics.Tensors/Tensor.xml ⚠️Warning View Details
xml/System.Numerics.Tensors/TensorPrimitives.xml ⚠️Warning View Details
xml/System.Runtime.Serialization/Formatter.xml ⚠️Warning View Details
xml/System.ServiceModel.Channels/HttpRequestMessageExtensionMethods.xml ⚠️Warning View Details
xml/System.ServiceModel.Channels/HttpRequestMessageProperty.xml ⚠️Warning View Details
xml/System.ServiceModel.Channels/HttpResponseMessageProperty.xml ⚠️Warning View Details
[xml/System.ServiceModel.Channels/NetFramingTransportChannelFactory1.xml](https://github.com/dotnet/dotnet-api-docs/blob/meai-evaluation/xml/System.ServiceModel.Channels/NetFramingTransportChannelFactory1.xml) ⚠️Warning View Details
xml/System.ServiceModel.Federation/IWSTrustChannelContract.xml ⚠️Warning View Details
xml/System.ServiceModel.Federation/WSTrustChannel.xml ⚠️Warning View Details
xml/System.ServiceModel.Federation/WSTrustChannelSecurityTokenProvider.xml ⚠️Warning View Details
xml/System.ServiceModel.Federation/WSTrustTokenParameters.xml ⚠️Warning View Details
xml/System.ServiceModel/HttpTransportSecurity.xml ⚠️Warning View Details
xml/Microsoft.Extensions.AI/AdditionalPropertiesDictionary.xml 💡Suggestion View Details

This comment lists only the first 25 files in the pull request.

xml/Microsoft.Extensions.AI/ChatClientBuilder.xml

  • [Warning: file-not-redirected - See documentation] File xml/Microsoft.Extensions.AI/ChatClientBuilder.xml with URL /dotnet/api/microsoft.extensions.ai.chatclientbuilder.services was deleted without redirection. To avoid broken links, add a redirection.

xml/Microsoft.Extensions.AI/ChatClientBuilderServiceCollectionExtensions.xml

  • [Warning: file-not-redirected - See documentation] File xml/Microsoft.Extensions.AI/ChatClientBuilderServiceCollectionExtensions.xml with URL /dotnet/api/microsoft.extensions.ai.chatclientbuilderservicecollectionextensions was deleted without redirection. To avoid broken links, add a redirection.
  • [Warning: file-not-redirected - See documentation] File xml/Microsoft.Extensions.AI/ChatClientBuilderServiceCollectionExtensions.xml with URL /dotnet/api/microsoft.extensions.ai.chatclientbuilderservicecollectionextensions.addchatclient was deleted without redirection. To avoid broken links, add a redirection.
  • [Warning: file-not-redirected - See documentation] File xml/Microsoft.Extensions.AI/ChatClientBuilderServiceCollectionExtensions.xml with URL /dotnet/api/microsoft.extensions.ai.chatclientbuilderservicecollectionextensions.addkeyedchatclient was deleted without redirection. To avoid broken links, add a redirection.

xml/Microsoft.Extensions.AI/ConfigureOptionsChatClientBuilderExtensions.xml

  • [Warning: file-not-redirected - See documentation] File xml/Microsoft.Extensions.AI/ConfigureOptionsChatClientBuilderExtensions.xml with URL /dotnet/api/microsoft.extensions.ai.configureoptionschatclientbuilderextensions.usechatoptions was deleted without redirection. To avoid broken links, add a redirection.

xml/Microsoft.Extensions.AI/ConfigureOptionsEmbeddingGeneratorBuilderExtensions.xml

  • [Warning: file-not-redirected - See documentation] File xml/Microsoft.Extensions.AI/ConfigureOptionsEmbeddingGeneratorBuilderExtensions.xml with URL /dotnet/api/microsoft.extensions.ai.configureoptionsembeddinggeneratorbuilderextensions.useembeddinggenerationoptions was deleted without redirection. To avoid broken links, add a redirection.

xml/Microsoft.Extensions.AI/EmbeddingGeneratorBuilder`2.xml

  • [Warning: file-not-redirected - See documentation] File xml/Microsoft.Extensions.AI/EmbeddingGeneratorBuilder`2.xml with URL /dotnet/api/microsoft.extensions.ai.embeddinggeneratorbuilder-2.services was deleted without redirection. To avoid broken links, add a redirection.

xml/Microsoft.Extensions.AI/EmbeddingGeneratorBuilderServiceCollectionExtensions.xml

  • [Warning: file-not-redirected - See documentation] File xml/Microsoft.Extensions.AI/EmbeddingGeneratorBuilderServiceCollectionExtensions.xml with URL /dotnet/api/microsoft.extensions.ai.embeddinggeneratorbuilderservicecollectionextensions was deleted without redirection. To avoid broken links, add a redirection.
  • [Warning: file-not-redirected - See documentation] File xml/Microsoft.Extensions.AI/EmbeddingGeneratorBuilderServiceCollectionExtensions.xml with URL /dotnet/api/microsoft.extensions.ai.embeddinggeneratorbuilderservicecollectionextensions.addembeddinggenerator was deleted without redirection. To avoid broken links, add a redirection.
  • [Warning: file-not-redirected - See documentation] File xml/Microsoft.Extensions.AI/EmbeddingGeneratorBuilderServiceCollectionExtensions.xml with URL /dotnet/api/microsoft.extensions.ai.embeddinggeneratorbuilderservicecollectionextensions.addkeyedembeddinggenerator was deleted without redirection. To avoid broken links, add a redirection.

xml/Microsoft.Extensions.AI/OpenAIRealtimeExtensions.xml

  • Line 0, Column 0: [Warning: xref-not-found - See documentation] Cross reference not found: 'OpenAI.RealtimeConversation.ConversationFunctionTool'.
  • Line 0, Column 0: [Warning: xref-not-found - See documentation] Cross reference not found: 'OpenAI.RealtimeConversation.RealtimeConversationClient'.
  • Line 0, Column 0: [Warning: xref-not-found - See documentation] Cross reference not found: 'OpenAI.RealtimeConversation.ConversationFunctionTool'.
  • Line 0, Column 0: [Warning: xref-not-found - See documentation] Cross reference not found: 'OpenAI.RealtimeConversation.ConversationFunctionTool'.
  • Line 0, Column 0: [Warning: xref-not-found - See documentation] Cross reference not found: 'OpenAI.RealtimeConversation.RealtimeConversationClient'.
  • Line 0, Column 0: [Warning: xref-not-found - See documentation] Cross reference not found: 'OpenAI.RealtimeConversation.RealtimeConversationSession'.
  • Line 0, Column 0: [Warning: xref-not-found - See documentation] Cross reference not found: 'OpenAI.RealtimeConversation.RealtimeConversationSession'.
  • Line 0, Column 0: [Warning: xref-not-found - See documentation] Cross reference not found: 'OpenAI.RealtimeConversation.ConversationUpdate'.
  • Line 0, Column 0: [Warning: xref-not-found - See documentation] Cross reference not found: 'OpenAI.RealtimeConversation.ConversationUpdate'.
  • Line 0, Column 0: [Warning: xref-not-found - See documentation] Cross reference not found: 'OpenAI.RealtimeConversation.RealtimeConversationSession'.

xml/Microsoft.Extensions.DependencyInjection/OptionsServiceCollectionExtensions.xml

  • Line 0, Column 0: [Warning: disallowed-html-tag - See documentation] HTML tag 'seealso' isn't allowed. Replace it with approved Markdown or escape the brackets if the content is a placeholder.
  • Line 0, Column 0: [Warning: disallowed-html-tag - See documentation] HTML tag 'seealso' isn't allowed. Replace it with approved Markdown or escape the brackets if the content is a placeholder.
  • Line 0, Column 0: [Warning: disallowed-html-tag - See documentation] HTML tag 'seealso' isn't allowed. Replace it with approved Markdown or escape the brackets if the content is a placeholder.
  • Line 0, Column 0: [Warning: disallowed-html-tag - See documentation] HTML tag 'seealso' isn't allowed. Replace it with approved Markdown or escape the brackets if the content is a placeholder.
  • Line 0, Column 0: [Warning: disallowed-html-tag - See documentation] HTML tag 'seealso' isn't allowed. Replace it with approved Markdown or escape the brackets if the content is a placeholder.

This comment lists only the first 25 errors (including error/warning/suggestion) in the pull request.
For more details, please refer to the build report.

Note: Your PR may contain errors or warnings or suggestions unrelated to the files you changed. This happens when external dependencies like GitHub alias, Microsoft alias, cross repo links are updated. Please use these instructions to resolve them.

For any questions, please:

@shyamnamboodiripad
Copy link

@shyamnamboodiripad Sure I will rerun with the new version.

@gewarren The latest changes for MEAI.Evaluation look great!👍🏾I have approved the PR.

I am not sure whether it is by design that docs for other unrelated BCL and Microsoft.Extensions assemblies (including MEAI) are also being refreshed in this PR, but please feel free to merge if everything else looks good on your side. Thanks so much!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants