Troubleshooting - Deploying installer-provisioned clusters on bare metal | Installing

About
- Welcome
- Legal notice
What's new?
- New features and enhancements
- Deprecated features
Release notes
- OpenShift Container Platform 4.6 release notes
- Versioning policy
Architecture
- Product architecture
- Installation and update
- The control plane
- Understanding OKD development
- Fedora CoreOS
- Using ArgoCD
Installing
- Mirroring images for a disconnected installation
- Installing on AWS
- Installing on Azure
- Installing on GCP
- Installing on bare metal
- Deploying installer-provisioned clusters on bare metal
- Installing on OpenStack
- Installing on RHV
- Installing on vSphere
- Installing on VMC
- Installing on any platform
  - Installing a cluster on any platform
- Installation configuration
- Validating an installation
- Troubleshooting installation issues
Updating clusters
- Understanding the OpenShift Update Service
- Installing and configuring the OpenShift Update Service
- Updating a cluster between minor versions
- Updating a cluster within a minor version from the web console
- Updating a cluster within a minor version by using the CLI
Post-installation configuration
- Post-installation configuration overview
- Machine configuration tasks
- Cluster tasks
- Node tasks
- Network configuration
- Storage configuration
- Preparing for users
- Configuring alert notifications
Web console
- Accessing the web console
- Viewing cluster information
- Configuring the web console
- Customizing the web console
- Developer perspective
- Web terminal
- Disabling the web console
CLI tools
- CLI tools overview
- OpenShift CLI (oc)
- Developer CLI (odo)
- Helm CLI
  - Getting started with Helm
  - Configuring custom Helm chart repositories
- Knative CLI (kn) for use with OpenShift Serverless
- opm CLI
Security and compliance
- Security and compliance overview
- Container security
- Configuring certificates
- Certificate types and descriptions
- Compliance Operator
- File Integrity Operator
- Viewing audit logs
- Configuring the audit log policy
- Configuring TLS security profiles
- Configuring seccomp profiles
- Allowing JavaScript-based access to the API server from additional hosts
- Encrypting etcd data
- Scanning pods for vulnerabilities
Authentication and authorization
- Understanding authentication
- Configuring the internal OAuth server
- Configuring OAuth clients
- Understanding identity provider configuration
- Configuring identity providers
- Using RBAC to define and apply permissions
- Removing the kubeadmin user
- Understanding and creating service accounts
- Using service accounts in applications
- Using a service account as an OAuth client
- Scoping tokens
- Using bound service account tokens
- Managing security context constraints
- Impersonating the system:admin user
- Syncing LDAP groups
Networking
- Understanding networking
- Accessing hosts
- Understanding the Cluster Network Operator
- Understanding the DNS Operator
- Understanding the Ingress Operator
- Configuring the node port service range
- Using SCTP
- Configuring PTP hardware
- Network policy
- Multiple networks
- Hardware networks
- OpenShift SDN default CNI network provider
- OVN-Kubernetes default CNI network provider
- Configuring Routes
  - Route configuration
  - Secured routes
- Configuring ingress cluster traffic
- Configuring the cluster-wide proxy
- Configuring a custom PKI
- Load balancing on OpenStack
- Associating secondary interfaces metrics to network attachments
Storage
- Understanding ephemeral storage
- Understanding persistent storage
- Configuring persistent storage
- Using Container Storage Interface (CSI)
- Expanding persistent volumes
- Dynamic provisioning
Registry
- Overview
- Image Registry Operator in OKD
- Setting up and configuring the registry
- Registry options
- Accessing the registry
- Exposing the registry
Operators
- Understanding Operators
- User tasks
  - Creating applications from installed Operators
  - Installing Operators in your namespace
- Administrator tasks
- Developing Operators
- Red Hat Operators reference
Builds
- Understanding image builds
- Understanding build configurations
- Creating build inputs
- Managing build output
- Using build strategies
- Custom image builds with Buildah
- Performing basic builds
- Triggering and modifying builds
- Performing advanced builds
- Using Red Hat subscriptions in builds
- Securing builds by strategy
- Build configuration resources
- Troubleshooting builds
- Setting up additional trusted certificate authorities for builds
Images
- Configuring the Cluster Samples Operator
- Using the Cluster Samples Operator with an alternate registry
- Understanding containers, images, and imagestreams
- Creating images
- Managing images
- Managing image streams
- Using image streams with Kubernetes resources
- Triggering updates on image stream changes
- Image configuration resources
- Using templates
- Using Ruby on Rails
- Using images
Applications
- Projects
- Application life cycle management
- Deployments
- Quotas
  - Resource quotas per project
  - Resource quotas across multiple projects
- Using config maps with applications
- Monitoring project and application metrics using the Developer perspective
- Monitoring application health
- Idling applications
- Pruning objects to reclaim resources
- Using the Red Hat Marketplace
Machine management
- Creating machine sets
- Manually scaling a machine set
- Modifying a machine set
- Deleting a machine
- Applying autoscaling to a cluster
- Creating infrastructure machine sets
- User-provisioned infrastructure
- Deploying machine health checks
Nodes
- Overview of nodes
- Working with pods
- Controlling pod placement onto nodes (scheduling)
- Using Jobs and DaemonSets
  - Running background tasks on nodes automatically with daemonsets
  - Running tasks in pods using jobs
- Working with nodes
- Working with containers
- Working with clusters
Windows Container Support for OpenShift
- Windows Container Support for OpenShift release notes
- Understanding Windows container workloads
- Enabling Windows container workloads
- Creating Windows MachineSet objects
  - Creating a Windows MachineSet object on AWS
  - Creating a Windows MachineSet object on Azure
- Scheduling Windows container workloads
- Windows node upgrades
- Removing Windows nodes
- Disabling Windows container workloads
Logging
- About cluster logging
- Installing cluster logging
- Configuring your cluster logging deployment
- Viewing logs for a specific resource
- Viewing cluster logs in Kibana
- Forwarding logs to third party systems
- Collecting and storing Kubernetes events
- Updating cluster logging
- Viewing cluster dashboards
- Troubleshooting cluster logging
- Uninstalling cluster logging
- Exported fields
Monitoring
- Understanding the monitoring stack
- Configuring the monitoring stack
- Enabling monitoring for user-defined projects
- Managing metrics
- Managing alerts
- Reviewing monitoring dashboards
- Accessing third-party UIs
- Exposing custom application metrics for autoscaling
- Troubleshooting monitoring issues
Metering
- About metering
- Installing metering
- Upgrading metering
- Configuring metering
- Reports
  - About reports
  - Storage Locations
- Using metering
- Examples of using metering
- Troubleshooting and debugging
- Uninstalling metering
Scalability and performance
- Recommended installation practices
- Recommended host practices
- Recommended cluster scaling practices
- Using the Node Tuning Operator
- Using Cluster Loader
- Using CPU Manager
- Using Topology Manager
- Scaling the Cluster Monitoring Operator
- Planning your environment according to object maximums
- Optimizing storage
- Optimizing routing
- Optimizing networking
- Managing bare metal hosts
- What huge pages do and how they are consumed by apps
- Performance Addon Operator for low latency nodes
- Optimizing data plane performance with Intel devices
Backup and restore
- Shutting down a cluster gracefully
- Restarting a cluster gracefully
- Control plane backup and restore
Migrating from version 3 to 4
- About migrating from OKD 3 to 4
- Differences between OKD 3 and 4
- Planning considerations
- About MTC
- Installing MTC
- Installing MTC in a restricted network environment
- Upgrading MTC
- Premigration checklists
- Migrating your applications
- Advanced migration options
- Troubleshooting
Migration Toolkit for Containers
- About MTC
- Installing MTC
- Installing MTC in a restricted network environment
- Upgrading MTC
- Premigration checklists
- Migrating your applications
- Advanced migration options
- Troubleshooting
API reference
- Editing kubelet log level verbosity and gathering logs
- API list
- Common object reference
  - Index
- Authorization APIs
  - About Authorization APIs
  - LocalResourceAccessReview [authorization.openshift.io/v1]
  - LocalSubjectAccessReview [authorization.openshift.io/v1]
  - ResourceAccessReview [authorization.openshift.io/v1]
  - SelfSubjectRulesReview [authorization.openshift.io/v1]
  - SubjectAccessReview [authorization.openshift.io/v1]
  - SubjectRulesReview [authorization.openshift.io/v1]
  - TokenReview [authentication.k8s.io/v1]
  - LocalSubjectAccessReview [authorization.k8s.io/v1]
  - SelfSubjectAccessReview [authorization.k8s.io/v1]
  - SelfSubjectRulesReview [authorization.k8s.io/v1]
  - SubjectAccessReview [authorization.k8s.io/v1]
- Autoscale APIs
  - About Autoscale APIs
  - ClusterAutoscaler [autoscaling.openshift.io/v1]
  - MachineAutoscaler [autoscaling.openshift.io/v1beta1]
  - HorizontalPodAutoscaler [autoscaling/v1]
- Config APIs
  - About Config APIs
  - APIServer [config.openshift.io/v1]
  - Authentication [config.openshift.io/v1]
  - Build [config.openshift.io/v1]
  - ClusterOperator [config.openshift.io/v1]
  - ClusterVersion [config.openshift.io/v1]
  - Console [config.openshift.io/v1]
  - DNS [config.openshift.io/v1]
  - FeatureGate [config.openshift.io/v1]
  - HelmChartRepository [helm.openshift.io/v1beta1]
  - Image [config.openshift.io/v1]
  - Infrastructure [config.openshift.io/v1]
  - Ingress [config.openshift.io/v1]
  - Network [config.openshift.io/v1]
  - OAuth [config.openshift.io/v1]
  - OperatorHub [config.openshift.io/v1]
  - Project [config.openshift.io/v1]
  - Proxy [config.openshift.io/v1]
  - Scheduler [config.openshift.io/v1]
- Console APIs
  - About Console APIs
  - ConsoleCLIDownload [console.openshift.io/v1]
  - ConsoleExternalLogLink [console.openshift.io/v1]
  - ConsoleLink [console.openshift.io/v1]
  - ConsoleNotification [console.openshift.io/v1]
  - ConsoleYAMLSample [console.openshift.io/v1]
- Extension APIs
  - About Extension APIs
  - APIService [apiregistration.k8s.io/v1]
  - CustomResourceDefinition [apiextensions.k8s.io/v1]
  - MutatingWebhookConfiguration [admissionregistration.k8s.io/v1]
  - ValidatingWebhookConfiguration [admissionregistration.k8s.io/v1]
- Image APIs
  - About Image APIs
  - Image [image.openshift.io/v1]
  - ImageSignature [image.openshift.io/v1]
  - ImageStreamImage [image.openshift.io/v1]
  - ImageStreamImport [image.openshift.io/v1]
  - ImageStreamMapping [image.openshift.io/v1]
  - ImageStream [image.openshift.io/v1]
  - ImageStreamTag [image.openshift.io/v1]
  - ImageTag [image.openshift.io/v1]
- Machine APIs
  - About Machine APIs
  - ContainerRuntimeConfig [machineconfiguration.openshift.io/v1]
  - ControllerConfig [machineconfiguration.openshift.io/v1]
  - KubeletConfig [machineconfiguration.openshift.io/v1]
  - MachineConfigPool [machineconfiguration.openshift.io/v1]
  - MachineConfig [machineconfiguration.openshift.io/v1]
  - MachineHealthCheck [machine.openshift.io/v1beta1]
  - Machine [machine.openshift.io/v1beta1]
  - MachineSet [machine.openshift.io/v1beta1]
- Metadata APIs
  - About Metadata APIs
  - Binding [core/v1]
  - ComponentStatus [core/v1]
  - ConfigMap [core/v1]
  - ControllerRevision [apps/v1]
  - Event [events.k8s.io/v1]
  - Event [core/v1]
  - Lease [coordination.k8s.io/v1]
  - Namespace [core/v1]
- Monitoring APIs
  - About Monitoring APIs
  - Alertmanager [monitoring.coreos.com/v1]
  - PodMonitor [monitoring.coreos.com/v1]
  - Probe [monitoring.coreos.com/v1]
  - Prometheus [monitoring.coreos.com/v1]
  - PrometheusRule [monitoring.coreos.com/v1]
  - ServiceMonitor [monitoring.coreos.com/v1]
  - ThanosRuler [monitoring.coreos.com/v1]
- Network APIs
  - About Network APIs
  - ClusterNetwork [network.openshift.io/v1]
  - Endpoints [core/v1]
  - EndpointSlice [discovery.k8s.io/v1beta1]
  - EgressNetworkPolicy [network.openshift.io/v1]
  - HostSubnet [network.openshift.io/v1]
  - Ingress [networking.k8s.io/v1]
  - IngressClass [networking.k8s.io/v1]
  - IPPool [whereabouts.cni.cncf.io/v1alpha1]
  - NetNamespace [network.openshift.io/v1]
  - NetworkAttachmentDefinition [k8s.cni.cncf.io/v1]
  - NetworkPolicy [networking.k8s.io/v1]
  - Route [route.openshift.io/v1]
  - Service [core/v1]
- Node APIs
  - About Node APIs
  - Node [core/v1]
  - Profile [tuned.openshift.io/v1]
  - RuntimeClass [node.k8s.io/v1beta1]
  - Tuned [tuned.openshift.io/v1]
- OAuth APIs
  - About OAuth APIs
  - OAuthAccessToken [oauth.openshift.io/v1]
  - OAuthAuthorizeToken [oauth.openshift.io/v1]
  - OAuthClientAuthorization [oauth.openshift.io/v1]
  - OAuthClient [oauth.openshift.io/v1]
- Operator APIs
  - About Operator APIs
  - Authentication [operator.openshift.io/v1]
  - CloudCredential [operator.openshift.io/v1]
  - ClusterCSIDriver [operator.openshift.io/v1]
  - Console [operator.openshift.io/v1]
  - Config [operator.openshift.io/v1]
  - Config [imageregistry.operator.openshift.io/v1]
  - Config [samples.operator.openshift.io/v1]
  - CSISnapshotController [operator.openshift.io/v1]
  - DNS [operator.openshift.io/v1]
  - DNSRecord [ingress.operator.openshift.io/v1]
  - Etcd [operator.openshift.io/v1]
  - ImageContentSourcePolicy [operator.openshift.io/v1alpha1]
  - ImagePruner [imageregistry.operator.openshift.io/v1]
  - IngressController [operator.openshift.io/v1]
  - KubeAPIServer [operator.openshift.io/v1]
  - KubeControllerManager [operator.openshift.io/v1]
  - KubeScheduler [operator.openshift.io/v1]
  - KubeStorageVersionMigrator [operator.openshift.io/v1]
  - Network [operator.openshift.io/v1]
  - OpenShiftAPIServer [operator.openshift.io/v1]
  - OpenShiftControllerManager [operator.openshift.io/v1]
  - OperatorPKI [network.operator.openshift.io/v1]
  - ServiceCA [operator.openshift.io/v1]
  - Storage [operator.openshift.io/v1]
- OperatorHub APIs
  - About OperatorHub APIs
  - CatalogSource [operators.coreos.com/v1alpha1]
  - ClusterServiceVersion [operators.coreos.com/v1alpha1]
  - InstallPlan [operators.coreos.com/v1alpha1]
  - OperatorGroup [operators.coreos.com/v1]
  - PackageManifest [packages.operators.coreos.com/v1]
  - Subscription [operators.coreos.com/v1alpha1]
- Policy APIs
  - About Policy APIs
  - PodDisruptionBudget [policy/v1beta1]
- Project APIs
  - About Project APIs
  - Project [project.openshift.io/v1]
  - ProjectRequest [project.openshift.io/v1]
- Provisioning APIs
  - About Provisioning APIs
  - BareMetalHost [metal3.io/v1alpha1]
  - Provisioning [metal3.io/v1alpha1]
- RBAC APIs
  - About RBAC APIs
  - ClusterRoleBinding [rbac.authorization.k8s.io/v1]
  - ClusterRole [rbac.authorization.k8s.io/v1]
  - RoleBinding [rbac.authorization.k8s.io/v1]
  - Role [rbac.authorization.k8s.io/v1]
- Role APIs
  - About Role APIs
  - ClusterRoleBinding [authorization.openshift.io/v1]
  - ClusterRole [authorization.openshift.io/v1]
  - RoleBindingRestriction [authorization.openshift.io/v1]
  - RoleBinding [authorization.openshift.io/v1]
  - Role [authorization.openshift.io/v1]
- Schedule and quota APIs
  - About Schedule and quota APIs
  - AppliedClusterResourceQuota [quota.openshift.io/v1]
  - ClusterResourceQuota [quota.openshift.io/v1]
  - FlowSchema [flowcontrol.apiserver.k8s.io/v1alpha1]
  - LimitRange [core/v1]
  - PriorityClass [scheduling.k8s.io/v1]
  - PriorityLevelConfiguration [flowcontrol.apiserver.k8s.io/v1alpha1]
  - ResourceQuota [core/v1]
- Security APIs
  - About Security APIs
  - CertificateSigningRequest [certificates.k8s.io/v1]
  - CredentialsRequest [cloudcredential.openshift.io/v1]
  - PodSecurityPolicyReview [security.openshift.io/v1]
  - PodSecurityPolicySelfSubjectReview [security.openshift.io/v1]
  - PodSecurityPolicySubjectReview [security.openshift.io/v1]
  - RangeAllocation [security.openshift.io/v1]
  - Secret [core/v1]
  - SecurityContextConstraints [security.openshift.io/v1]
  - ServiceAccount [core/v1]
- Storage APIs
  - About Storage APIs
  - CSIDriver [storage.k8s.io/v1]
  - CSINode [storage.k8s.io/v1]
  - PersistentVolumeClaim [core/v1]
  - StorageClass [storage.k8s.io/v1]
  - StorageState [migration.k8s.io/v1alpha1]
  - StorageVersionMigration [migration.k8s.io/v1alpha1]
  - VolumeAttachment [storage.k8s.io/v1]
  - VolumeSnapshot [snapshot.storage.k8s.io/v1beta1]
  - VolumeSnapshotClass [snapshot.storage.k8s.io/v1beta1]
  - VolumeSnapshotContent [snapshot.storage.k8s.io/v1beta1]
- Template APIs
  - About Template APIs
  - BrokerTemplateInstance [template.openshift.io/v1]
  - PodTemplate [core/v1]
  - Template [template.openshift.io/v1]
  - TemplateInstance [template.openshift.io/v1]
- User and group APIs
  - About User and group APIs
  - Group [user.openshift.io/v1]
  - Identity [user.openshift.io/v1]
  - UserIdentityMapping [user.openshift.io/v1]
  - User [user.openshift.io/v1]
- Workloads APIs
  - About Workloads APIs
  - BuildConfig [build.openshift.io/v1]
  - Build [build.openshift.io/v1]
  - CronJob [batch/v1beta1]
  - DaemonSet [apps/v1]
  - Deployment [apps/v1]
  - DeploymentConfig [apps.openshift.io/v1]
  - Job [batch/v1]
  - Pod [core/v1]
  - ReplicationController [core/v1]
  - PersistentVolume [core/v1]
  - ReplicaSet [apps/v1]
  - StatefulSet [apps/v1]

Troubleshooting the installer workflow
Troubleshooting install-config.yaml
Bootstrap VM issues
- Bootstrap VM cannot boot up the cluster nodes
- Inspecting logs
Cluster nodes will not PXE boot
The API is not accessible
Cleaning up previous installations
Issues with creating the registry
Miscellaneous issues
Reviewing the installation

Troubleshooting the installer workflow

Prior to troubleshooting the installation environment, it is critical to understand the overall flow of the installer-provisioned installation on bare metal. The diagrams below provide a troubleshooting flow with a step-by-step breakdown for the environment.

Flow-Diagram-1

Workflow 1 of 4 illustrates a troubleshooting workflow when the install-config.yaml file has errors or the Fedora CoreOS (FCOS) images are inaccessible. Troubleshooting suggestions can be found at Troubleshooting install-config.yaml.

Flow-Diagram-2

Workflow 2 of 4 illustrates a troubleshooting workflow for bootstrap VM issues, bootstrap VMs that cannot boot up the cluster nodes, and inspecting logs. When installing a OKD cluster without the provisioning network, this workflow does not apply.