Skip to content
GitLab
Commit 6e86f1ea authored by Vlad Zakharchenko's avatar Vlad Zakharchenko
Browse files

Added documentation on the OpenIMF platform

parents
Hide whitespace changes
Inline Side-by-side
CODE_OF_CONDUCT.md 0 → 100644
View file @ 6e86f1ea
# Contributor Covenant Code of Conduct
## Our Pledge
We as members, contributors, and leaders pledge to make participation in our
community a harassment-free experience for everyone, regardless of age, body
size, visible or invisible disability, ethnicity, sex characteristics, gender
identity and expression, level of experience, education, socio-economic status,
nationality, personal appearance, race, religion, or sexual identity
and orientation.
We pledge to act and interact in ways that contribute to an open, welcoming,
diverse, inclusive, and healthy community.
## Our Standards
Examples of behavior that contributes to a positive environment for our
community include:
* Demonstrating empathy and kindness toward other people
* Being respectful of differing opinions, viewpoints, and experiences
* Giving and gracefully accepting constructive feedback
* Accepting responsibility and apologizing to those affected by our mistakes,
and learning from the experience
* Focusing on what is best not just for us as individuals, but for the
overall community
Examples of unacceptable behavior include:
* The use of sexualized language or imagery, and sexual attention or
advances of any kind
* Trolling, insulting or derogatory comments, and personal or political attacks
* Public or private harassment
* Publishing others' private information, such as a physical or email
address, without their explicit permission
* Other conduct which could reasonably be considered inappropriate in a
professional setting
## Enforcement Responsibilities
Community leaders are responsible for clarifying and enforcing our standards of
acceptable behavior and will take appropriate and fair corrective action in
response to any behavior that they deem inappropriate, threatening, offensive,
or harmful.
Community leaders have the right and responsibility to remove, edit, or reject
comments, commits, code, wiki edits, issues, and other contributions that are
not aligned to this Code of Conduct, and will communicate reasons for moderation
decisions when appropriate.
## Scope
This Code of Conduct applies within all community spaces, and also applies when
an individual is officially representing the community in public spaces.
Examples of representing our community include using an official e-mail address,
posting via an official social media account, or acting as an appointed
representative at an online or offline event.
## Enforcement
Instances of abusive, harassing, or otherwise unacceptable behavior may be
reported to the community leaders responsible for enforcement at
[INSERT CONTACT METHOD].
All complaints will be reviewed and investigated promptly and fairly.
All community leaders are obligated to respect the privacy and security of the
reporter of any incident.
## Enforcement Guidelines
Community leaders will follow these Community Impact Guidelines in determining
the consequences for any action they deem in violation of this Code of Conduct:
### 1. Correction
**Community Impact**: Use of inappropriate language or other behavior deemed
unprofessional or unwelcome in the community.
**Consequence**: A private, written warning from community leaders, providing
clarity around the nature of the violation and an explanation of why the
behavior was inappropriate. A public apology may be requested.
### 2. Warning
**Community Impact**: A violation through a single incident or series
of actions.
**Consequence**: A warning with consequences for continued behavior. No
interaction with the people involved, including unsolicited interaction with
those enforcing the Code of Conduct, for a specified period of time. This
includes avoiding interactions in community spaces as well as external channels
like social media. Violating these terms may lead to a temporary or
permanent ban.
### 3. Temporary Ban
**Community Impact**: A serious violation of community standards, including
sustained inappropriate behavior.
**Consequence**: A temporary ban from any sort of interaction or public
communication with the community for a specified period of time. No public or
private interaction with the people involved, including unsolicited interaction
with those enforcing the Code of Conduct, is allowed during this period.
Violating these terms may lead to a permanent ban.
### 4. Permanent Ban
**Community Impact**: Demonstrating a pattern of violation of community
standards, including sustained inappropriate behavior, harassment of an
individual, or aggression toward or disparagement of classes of individuals.
**Consequence**: A permanent ban from any sort of public interaction within
the community.
## Attribution
This Code of Conduct is adapted from the [Contributor Covenant][homepage],
version 2.0, available at
https://www.contributor-covenant.org/version/2/0/code_of_conduct.html.
Community Impact Guidelines were inspired by [Mozilla's code of conduct
enforcement ladder](https://github.com/mozilla/diversity).
[homepage]: https://www.contributor-covenant.org
For answers to common questions about this code of conduct, see the FAQ at
https://www.contributor-covenant.org/faq. Translations are available at
https://www.contributor-covenant.org/translations.
\ No newline at end of file
CONTRIBUTING.md 0 → 100644
View file @ 6e86f1ea
# Welcome
Welcome to OpenIMF!
- [Before you get started](#before-you-get-started)
- [Code of Conduct](#code-of-conduct)
- [Community Expectations](#community-expectations)
- [Getting started](#getting-started)
- [Your First Contribution](#your-first-contribution)
- [Find something to work on](#find-something-to-work-on)
- [Find a good first topic](#find-a-good-first-topic)
- [Work on an Issue](#work-on-an-issue)
- [File an Issue](#file-an-issue)
- [Contributor Workflow](#contributor-workflow)
- [Creating Pull Requests](#creating-pull-requests)
- [Code Review](#code-review)
- [Testing](#testing)
# Before you get started
## Code of Conduct
Please make sure to read and observe our [Code of Conduct](http://mpegx.int-evry.fr/software/MPEG-Public/PCC/V-PCC_decoder/CODE_OF_CONDUCT.md).
## Community Expectations
OpenIMF is a community project driven by its community which strives to promote a healthy, friendly and productive environment.
The goal of the community is to develop a platform for delivery of the immersive media over the network. To build a platform at such scale requires the support of a community with similar aspirations.
- See [Community Membership](./community-membership.md) for a list of various community roles. With gradual contributions, one can move up in the chain.
# Getting started
- Fork the repository on GitLab
- Read the [developer guide](../setup/develop_OpenIMF.md) for build instructions.
# Your First Contribution
We will help you to contribute in different areas like filing issues, developing features, fixing critical bugs and getting your work reviewed and merged.
If you have questions about the development process, feel free to jump into our [Slack Channel](https://join.slack.com/t/OpenIMF) or join our [mailing list](https://groups.google.com/forum/#!forum/OpenIMF).
## Find something to work on
We are always in need of help, be it fixing documentation, reporting bugs or writing some code.
Look at places where you feel best coding practices aren't followed, code refactoring is needed or tests are missing.
Here is how you get started.
### Find a good first topic
There is currently only one [repository](http://mpegx.int-evry.fr/software/MPEG-Public/PCC/V-PCC_decoder) within the OpenIMF organization.
We can help new contributors who wish to work on such issues.
Another good way to contribute is to find a documentation improvement, such as a missing/broken link. Please see [Contributing](#contributing) below for the workflow.
#### Work on an issue
When you are willing to take on an issue, you can assign it to yourself. Just reply with `/assign` or `/assign @yourself` on an issue,
then the robot will assign the issue to you and your name will present at `Assignees` list.
### File an Issue
While we encourage everyone to contribute code, it is also appreciated when someone reports an issue.
Issues should be filed under the OpenIMF repository.
*Example:* a OpenIMF issue should be opened to [OpenIMF](http://mpegx.int-evry.fr/software/MPEG-Public/PCC/V-PCC_decoder/issues).
Please follow the prompted submission guidelines while opening an issue.
# Contributor Workflow
Please do not ever hesitate to ask a question or send a pull request.
This is a rough outline of what a contributor's workflow looks like:
- Create a topic branch from where to base the contribution. This is usually master.
- Make commits of logical units.
- Make sure commit messages are in the proper format (see below).
- Push changes in a topic branch to a personal fork of the repository.
- Submit a pull request to [OpenIMF](http://mpegx.int-evry.fr/software/MPEG-Public/PCC/V-PCC_decoder).
- The PR must receive an approval from two maintainers.
## Creating Pull Requests
Pull requests are often called simply "PR".
OpenIMF generally follows the standard [gitlab pull request](https://docs.gitlab.com/ee/topics/gitlab_flow.html) process.
In addition to the above process, a bot will begin applying structured labels to your PR.
The bot may also make some helpful suggestions for commands to run in your PR to facilitate review.
These `/command` options can be entered in comments to trigger auto-labeling and notifications.
Refer to its [command reference documentation](https://TBU).
## Code Review
To make it easier for your PR to receive reviews, consider the reviewers will need you to:
* follow [good coding guidelines](https://github.com/golang/go/wiki/CodeReviewComments).
* write [good commit messages](https://chris.beams.io/posts/git-commit/).
* break large changes into a logical series of smaller patches which individually make easily understandable changes, and in aggregate solve a broader issue.
* label PRs with appropriate reviewers: to do this read the messages the bot sends you to guide you through the PR process.
### Format of the commit message
We follow a rough convention for commit messages that is designed to answer two questions: what changed and why.
The subject line should feature the what and the body of the commit should describe the why.
```
scripts: add test codes for android decoder
this add some unit test codes to improve code coverage for android v-pcc decoder
Fixes #42
```
The format can be described more formally as follows:
```
<subsystem>: <what changed>
<BLANK LINE>
<why this change was made>
<BLANK LINE>
<footer>
```
The first line is the subject and should be no longer than 70 characters, the second line is always blank, and other lines should be wrapped at 80 characters. This allows the message to be easier to read on GitLab as well as in various git tools.
Note: if your pull request isn't getting enough attention, you can use the reach out on Slack to get help finding reviewers.
## Testing
There are multiple types of tests.
The location of the test code varies with type, as do the specifics of the environment needed to successfully run the test:
* Unit: These confirm that a particular function behaves as intended. Unit test source code can be found adjacent to the corresponding source code within a given package. These are easily run locally by any developer.
* Integration: These tests cover interactions of package components or interactions between OpenIMF components and OpenIMF core.
\ No newline at end of file
LICENSE 0 → 100644
View file @ 6e86f1ea
Copyright (c) 2020, Futurewei Technologies., Inc.
Licensed under the Apache License, Version 2.0 (the License);
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org//licenses//LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an AS IS BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
NOTICE 0 → 100644
View file @ 6e86f1ea
This product includes software developed at
The Apache Software Foundation (http://www.apache.org/).
This software uses libraries from the FFmpeg project under the LGPLv2.1
The FFmpeg binaries had been geenrated accoring to the guide provided at
https://trac.ffmpeg.org/wiki/CompilationGuide/MSVC
And configured to be using LGPL-2.1 license
./configure \
--toolchain=msvc \
--arch=x86_64 \
--enable-yasm \
--enable-asm \
--enable-shared \
--enable-static \
--enable-swscale \
--disable-programs \
--disable-gpl \
--disable-avdevice \
--disable-doc \
--disable-ffplay \
--disable-ffprobe \
--disable-ffmpeg \
--disable-bzlib \
--disable-libopenjpeg \
--disable-iconv \
--disable-zlib \
--target-os=win64 \
README.md 0 → 100644
View file @ 6e86f1ea
# OpenIMF v1.0
The project Open Immersive Media Framework is a cross platform reference decoder and reconstruction implementation of the ISO/IEC 23090-5 compression standard: MPEG-I part 5 "Video-based Point Cloud Compression"
## Building
Bash scripts can be use to build OpenIMF project:
- build.sh: build solutions.
- clear.sh: clear solututions.
### OSX
- mkdir build
- cd build
- cmake .. -G Xcode
- open the generated xcode project and build it
### Linux
- mkdir build
- cd build
- cmake ..
- make
### Windows
- md build
- cd build
- cmake .. -G "Visual Studio 15 2017 Win64"
- open the generated visual studio solution and build it
### Android
- open the android Studio
- import the project from porting/Android/PccDecoder
### External dependencies
| Software | License | Repo Link
|---|---|---|
| arcore-android-sdk-1.14.0 | Apache License 2.0 | https://developers.google.com/ar/develop
| ffmpeg-4.2.2-win64-dev-lgpl | LGPL v2.1 | https://github.com/FFmpeg/FFmpeg
| glew-2.1.0 | Khronos Group | https://github.com/nigels-com/glew
| glfw | Zlib | https://github.com/glfw/glfw
| glm | MIT | https://github.com/g-truc/glm
| stb | MIT | https://github.com/nothings/stb
### Running
The decoder requires an ISO/IEC 23090-5 comliant bitstream
Example bistreams can be located in the data folder
### Scripts
The helper tool to copy the immersive media streams to the test application folder on the android device
### Contributing
Please read the doc/getting-started/contribute.md for details on our code of conduct, and the process for submitting pull requests to us.
\ No newline at end of file
doc/LICENSE.txt 0 → 100644
View file @ 6e86f1ea
/* Copyright (c) 2020, Futurewei Technologies., Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
doc/design/openimf_core.md 0 → 100644
View file @ 6e86f1ea
# OpenIMF Core
The core of the OpenIMF allows corss platform implementation of
the main functionality for the immersive media decoding and
reconstruction.
The emerging dynamic point cloud compression standard
enables efficient transmission of volumetric media content
across the wireless networks.
The volumetric media enables a variety of rich user
experiences, including interacting with remote objects, telepresence,
and augmented reality (AR) experiences, where the
real world is augmented or enhanced by virtual objects. In
this paper, we present a solution for real-time decoding and
rendering of an immersive media using V-PCC compression
standard on a mobile devices, leveraging the mobile device
hardware including graphics processing unit (GPU) and
hardware video decoder to accelerate performance.
OpenIMF is a solution for real-time decoding and rendering of the
V-PCC content, leveraging the device hardware including GPU and
video decoder to accelerate performance.
OpenIMF supports AR experiences including the presentation of
v-pcc-decoded content super-imposed on the external environment
as seen through the camera.
The following diagram illustrates the __architecture__ of the OpenIMF.
![OpenIMF architecture](./figures/architecture.png)
Volumetric media such as dynamic point clouds enables
viewer to freely navigate in an augmented scene with a 6
degrees of freedom.
The philosophy of the V-PCC solution is to build volumetric
media delivery ecosystem based on the existing hardware
infrastructure. The technology is based on creating a set of
projections from 3d space to 2d space and using the variety
of video coding solutions to achieve the efficient compression
with an addition of the auxiliarry information that enables inverse
transformation from 2d projections to 3d volume
## V-PCC bitstream
The input bistream to the OpenIMF must comply to the ISO/IEC 23090-5.
Currenlt supported version of bitstreams shall be generated using the
TMC2v8.0 reference software.
The following diagram illustrates the __v-pcc__ bitstream components.
![v-pcc bitstream components](./figures/bitstreamSyntax.png)
## Decoding process
Well known problem component synchronization that is present in
the heterogeneous processing systems it transferring large amounts
of data, such as decoded images over the but from GPU to CPU memory
The following diagram illustrates the __decoder__ __pipeline__
of the OpenIMF.
![core](./figures/decoder_pipeline.png)
The architecture assumes that the video componenets
of the corresponding v-pcc sub-bitstream shall be decoded using the
hardware accelerated video decoder depending on the specific platform
The basic reconstruction process of the dynamic point cloud
data comprises of the basic 2d to 3d conversion based on atlas
information. The patch in projected image space is bounded
by a 2d bounding box with the coordinates defined in the atlas
data, the blocks with valid samples have a corresponding
non-zero value in the down-sampled occupancy map. The
geometry information provides the delta depth map for the
occupied samples of the patch while the attributes in this case
add the color to the reconstructed point cloud.
The volumetric information in a V-PCC bitstream is composed
of several sub bitstreams that represent atlas data, occupancy
map, geometry, attribute (color). The packing of projected
information is performed based on a block-basis.
![v-pcc bitstream](./figures/bitstream.png)
The input v-pcc bitstream is parsed by the preprocessing using and
the VUI, SEI, SPS, PPS and Atlas sub-bitstreams are processed using
the CPU. Ocuupancy map, Geometry and Attribute components are decoded
using the hardware module, and the result of the decoding is mapped
as a video surface directly in the GPU memory.
## Reconstruction process
The AR module processes the camera images to reconstruct and create
the volumetric scene, subsequently the AR module detects and tracks
planes and spatial anchor points. The tracked planes are presented
on the display, enabling the user to select a spatial anchor point:
which defines where to place the point cloud in the scene. After
the user selects a spatial anchor, the point cloud is displayed in
the scene using the anchor transformation matrix.
The following diagram illustrates the __v-pcc__ renderer architecture.
![v-pcc renderer](./figures/renderer.png)
The v-pcc rendering pipeline is composed of several steps:
* Initialize global variables
* Update transformation matrices
* Render the point cloud
On each frame, the point cloud is rendered with the following process
#### Generate a block to patch map.
This defines a mapping of blocks in atlas space to corresponding
patches.
#### Bind the GPU shader program.
The geometry shader program is binded. The geometry shader program
emits points directly on GPU, reducing CPU overhead and improving
performance. Alternatively, this concept can be achieved using a
compute shader approach.
#### Update GPU textures
The decoded images from the occupancy, geometry, and attribute
video streams are binded directly as GPU YUV (NV12) textures,
with no copy in either CPU or GPU space
#### Update GPU buffers
The uniform buffer: ubo, are updated with the following data:
frame width and height, occupancy resolution, and modelViewProj
matrix. The uniform buffer is stored in cache memory and contains
data that is constant for that frame.
#### Render the patches using single draw call
##### The uniform buffer is updated.
The uniform buffer is updated. The uniform buffer contains
the patch data as a global array, including canvas (atlas) to
patch matrices.
##### The vertex buffer is updated.
Where each entry defines a block in atlas space and
corresponding patch index
##### The draw call is issued to render the points.
The primitive count is set to be equal to the number of blocks.
Primitives (points) are emitted on GPU directly using geometry
shader or compute shader
#### GPU Rendering
##### Vertex Shader:
Copy the per-vertex data to geometry shader
##### Geometry Shader:
Enables instancing, where each instance emits multiple vertices.
This reduces CPU overhead.
Max # vertices emitted per geometry shader
invocation = occupancy_resolution * occupancy_resolution * num_layers.
Each invocation processes a sub-block in atlas space
doc/getting-started/community_membership.md 0 → 100644
View file @ 6e86f1ea
# OpenIMF Community Membership
**Note :** This document keeps changing based on the status and feedback of OpenIMF Community.
This document gives a brief overview of the OpenIMF community roles with the requirements and responsibilities associated with them.
| Role | Requirements | Responsibilities | Privileges |
| -----| ---------------- | ------------ | -------|
| [Member](#member) | Sponsor from 2 approvers, active in community, contributed to OpenIMF | Welcome and guide new contributors | OpenIMF GitLab organization Member |
| [Approver](#approver) | Sponsor from 2 maintainers, has good experience and knowledge of domain, actively contributed to code and review | Review and approve contributions from community members | Write access to specific packagies in relevant repository |
| [Maintainer](#maintainer) | Sponsor from 2 owners, shown good technical judgement in feature design/development and PR review | Participate in release planning and feature development/maintenance | Top level write access to relevant repository. Name entry in Maintainers file of the repository |
| [Owner](#owner) | Sponsor from 3 owners, helps drive the overall OpenIMF project | Drive the overall technical roadmap of the project and set priorities of activities in release planning | OpenImf GitLab organization Admin access |
**Note :** It is mandatory for all OpenIMF community members to follow OpenIMF [Code of Conduct].
## Member
Members are active participants in the community who contribute by authoring PRs,
reviewing issues/PRs or participate in community discussions on slack/mailing list.
### Requirements
- Sponsor from 2 approvers
- Enabled [two-factor authentication] on their GitLab account
- Actively contributed to the community. Contributions may include, but are not limited to:
- Authoring PRs
- Reviewing issues/PRs authored by other community members
- Participating in community discussions on slack/mailing list
- Participate in OpenIMF community meetings
### Responsibilities and privileges
- Member of the OpenIMF GitLab organization
- Can be assigned to issues and PRs and community members can also request their review
- Participate in assigned issues and PRs
- Welcome new contributors
- Guide new contributors to relevant docs/files
- Help/Motivate new members in contributing to OpenIMF
## Approver
Approvers are active members who have good experience and knowledge of the domain.
They have actively participated in the issue/PR reviews and have identified relevant issues during review.
### Requirements
- Sponsor from 2 maintainers
- Member for at least 2 months
- Have reviewed good number of PRs
- Have good codebase knowledge
### Responsibilities and Privileges
- Review code to maintain/improve code quality
- Acknowledge and work on review requests from community members
- May approve code contributions for acceptance related to relevant expertise
- Have 'write access' to specific packages inside a repo, enforced via bot
- Continue to contribute and guide other community members to contribute in OpenIMF project
## Maintainer
Maintainers are approvers who have shown good technical judgement in feature design/development in the past.
Has overall knowledge of the project and features in the project.
### Requirements
- Sponsor from 2 owners
- Approver for at least 2 months
- Nominated by a project owner
- Good technical judgement in feature design/development
### Responsibilities and privileges
- Participate in release planning